Re: [PATCH pve-firewall 3/3] fix #1965: firewall: cache cluster config

[Arthur Bied-Charreton <a.bied-charreton@proxmox.com>]

On Wed, Nov 04, 2026 at 05:36:44PM +0100, Stefan Hanreich wrote:
> On 11/4/26 3:20 PM, Arthur Bied-Charreton wrote:
> > Large firewall configs can take a noticeable amount of time to parse,
> > as noted by the user on the linked issue.
> > 
> > Register the config with cfs_register_file in order to avoid re-parsing
> > it unless cluster.fw changed.
> > 
> > Rule verification still depends on the SDN config, which can change
> > independently. A rule referencing an SDN ipset that did not exist when
> > it was parsed and cached is still stored with errors, even if that rule
> > becomes valid later (e.g. if the rule references 'foo-dhcp' and the
> > VNet 'foo' is created later).
> > 
> > For this reason, the cached config keeps the SDN config it was verified
> > against last time it was fully parsed. If the ipset names yielded by the
> > current SDN config differ from what is stored, all cluster and group
> > rules are re-verified against the new ones. The ccache cannot be
> > invalidated from pve-firewall, so after an SDN change this happens on
> > every load until cluster.fw is parsed again due to a new version or a
> > daemon restart.
> > 
> > Errors found on re-verification are logged to preserve the original
> > behavior.
> > 
> > Benchmark with two 25k-entry ipsets and 1050 rules (50 loads):
> > 
> >     without caching:
> >         first load 0.2447s, then avg 0.2509s (min 0.2301 max 0.2835)
> > 
> >     cached, not re-verified (steady state):
> >         first load 0.2694s, then avg 0.0157s (min 0.0131 max 0.0204)
> > 
> >     cached, re-verified (worst case, transient until next parse):
> >         avg 0.0721s (min 0.0657 max 0.1012)
> That is with eea1b88b9 applied already? In that case the gain seems
> marginal? Are there other possible configurations that would get
> a bigger speed up. Because the core problem seems to be addressed
> with eea1b88b9 alone and then we don't have the added logic of
> re-verifying / caching?
> 
Yes, this already includes eea1b88. Without that commit, the same
config parses in 1.66s on my test cluster.

I reproduced this user's configuration as described here [0] (with 
50/50 IPv6 and IPv4):

    - 29,430 lines 
    - Approximately 468 KB
    - 13 IPSet definitions
    - 29,225 IPSet entries
    - 26 Security Groups
    - 83 Security Group entries

The user reports ~7.3s response time for firewall endpoitns with 6.0.6,
which has the regression from 99343fe but not yet eea1b88, and ~1.5s on
their PVE 8 cluster, i.e. the pre-regression parser. On my (idle) test
cluster the reproduced config gives ~2s and ~0.5s, respectively, so it
seems like their hardware is ~3x slower. eea1b88 brings them back to the
PVE 8 numbers.

The cache addresses an issue that predates the regression eea1b88 fixes.
I do agree it seems marginal on my tests, but given a lot of our
customers seem to be running slower hardware, a large cluster config
might still be costing almost a second per 10 second firewall cycle,
unconditionally, with a config nearing the 1mb pmxcfs size limit.

> Are there other possible configurations that would get a bigger speed
> up.
Rules with both literal source and literal destination address are more
expensive to parse (got up to 1.986s avg on a config with ~18k rules).
IPv6 vs. IPv4 did not make any meaningful difference after eea1b88b9. 

I agree that the caching logic is a little scary here because of the SDN
re-verification. I have no particularly strong feelings about this given
the nftables config parser is a lot faster anyway. I just think the
numbers on my machine might not be reflecting the actual cost on typical
customer hardware. 

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=8261