From: Arthur Bied-Charreton <a.bied-charreton@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH container/docs/firewall/manager/network/qemu-server v7 00/28] handle dangling references when firewall objects go away
Date: Thu, 12 Nov 2026 16:10:17 +0100 [thread overview]
Message-ID: <20261112151045.577923-1-a.bied-charreton@proxmox.com> (raw)
Renaming or deleting a firewall object (an IPSet, alias or security
group) that rules still reference leaves those references dangling.
The firewall fails to parse the affected rules and drops them from the
generated ruleset. For security groups, the reference reaches the
backend, which creates empty chains on iptables, and causes the ruleset
to fail to load on nftables [1].
This is especially bad in the rename case, where one might reasonably
expect the references to follow the new object name and walk away
thinking the firewall rules are active.
This series makes the affected operations offer to deal with the
references instead of leaving them behind with the following API
options:
POST .../firewall/ipset dangling-references keep|disable|drop|rename
force-rename boolean
PUT .../firewall/aliases/{name} dangling-references keep|disable|drop|rename
force-rename boolean
POST /cluster/firewall/groups dangling-references keep|disable|drop|rename
force-rename boolean
DELETE .../ipset/{name} dangling-references keep|disable|drop
DELETE .../aliases/{name} dangling-references keep|disable|drop
DELETE /cluster/firewall/groups/{name} dangling-references keep|disable|drop
PUT /cluster/sdn dangling-ipset-references keep|disable|drop
DELETE /nodes/{node}/qemu/{vmid} dangling-ipset-references keep|disable|drop
DELETE /nodes/{node}/lxc/{vmid} dangling-ipset-references keep|disable|drop
Overview:
- 'rename' rewrites the references to point at the new name, 'disable'
disables referencing rules, and 'drop' removes them. An IPSet member
has no disabled state so those are also removed with 'disable'. 'keep'
laves the references as they are.
- For a cluster object, all downstream configs in the cluster (guest,
host and vnet firewalls) are updated as well, locking and saving each
one (as was already partly done for security group renames). On a
rename, the new object is persisted before its references are
rewritten because otherwise concurrent compilations may see references
to objects that do not exist yet.
- If a cluster-wide rename is interrupted partway, the object exists
under both names and the next attempt fails because the target name
already exists. Setting 'force-rename' skips that check so the rename
can be retried.
- The API keeps backwards compatible defaults ('keep' everywhere except
for POST /cluster/firewall/groups, which was already renaming by
default and will keep doing that). The UI intentionally defaults to
renaming/dropping dangling references, as that is the most intuitive
behavior and every changed/dropped rule is logged in the task log.
- Passing 'dangling-references' runs the rename/delete as a task for
better observability. Every change made to configs is logged. Requests
that do not pass it (i.e. every existing client) still run
synchronously as before.
- SDN apply and guest destroy run the reference updates after the
destroy/commit succeeded. A failure to update references shows up as
that task failling. If the reference update fails, the remaining
references are left dangling as they would be without this series. A
generic cleanup/retry for dangling references could be a follow-up, that
would probably need to be a separate endpoint for SDN ipsets.
- Guest destroy and SDN apply can change firewall configs that the
caller would technically need different permissions for. They do not
require additional permissions, see [0]. Detailed logs on which configs
have been updated go into a root@pam-owned log.
Known limitations:
- Each config is locked and saved on its own, so a cluster-wide rename
is not atomic, which leaves two race windows. This is pre-existing
behavior, already present in security group renames and manual edits
via the API/GUI.
a. A compilation that reads the clsuter config before the new name was
saved and a guest config after it was rewritten skips rules
referencing the object *for the current cycle*. Closing this would
need the compliation to detect mixed reads and retry, which I
deemed out of scope for this series.
b. A rule added to a config that was already updated keeps the old
name. AFAICT the only way to properly solve this would be holding
locks for *all* firewall configs while editing, which would stall all
edits for the duration of the rename and therefore does not seem like
a great option.
- An unscoped reference stays unscoped when renamed. If a guest defines
an object under the target name, the guest's references start resolving
to that object after a rename. Rewriting them with an explicit scope
would change configs that are (sadly) sometimes managed outside of the
API. Detecting and skipping such references did not seem worth the added
complexity for such a narrow case, so this is accepted for now. We plan
to address this with a breaking change in PVE 10.
- Recovering from an interrupted rename is not covered by unit tests as
that would mean mocking most of PVE::Firewall and PVE::Cluster. It was
tested end to end on a test cluster.
Pre-built packages on sani under:
abied-charreton/firewall-object-references/v7/
Build dependencies:
pve-container <- pve-firewall
qemu-server <- pve-firewall
pve-manager <- pve-docs
Runtime dependencies:
pve-network <- pve-firewall
qemu-server <- pve-firewall
pve-container <- pve-firewall
pve-manager <- pve-firewall
pve-manager <- libpve-network-api-perl
pve-manager <- qemu-server
pve-manager <- pve-container
Changes since [v6]:
- SDN apply: only touch firewall configs after a successful commit so a
failed commit does not leaves rules dropped
- guest destroy: Only touch firewall configs after successful destroy
so a failed destroy does not leavbe rules dropped
- Update references to SDN-generated IPSets (guest destroy + SDN apply)
in a separate task owned by root@pam so the details are only visible to
node auditors
- docs: document the separate task and the UI defaults for guest destroy
and SDN apply, clarify that tasks are only spawned when the new option
is passed and explain what happens when a rule references a removed
auto-generated ipset.
---
Changes since [v5]:
- Make 'force-rename' an extra boolean parameter so a retry can also be
done with 'disable' or 'drop', not only with 'rename'
- Only run renames/deletes as tasks when 'dangling-references' is passed
so existing clients keep the same behavior
- Check that an alias exists before deleting it
- Log reference changes only after successful saves
- Walk the vnet configs present on disk instead of the ones from the
running config
- Add monkey patches to unit tests to prevent them from walking the
live cluster
- Use single gettext string for dangling references hints
- New API options are now shared definitions in pve-firewall
- A failed reference update in SDN apply logs the warning in the task
log instead of only the node's journal
[v6] https://lore.proxmox.com/pve-devel/20261111131901.1791124-1-a.bied-charreton@proxmox.com/
[v5] https://lore.proxmox.com/pve-devel/20261103084328.203492-1-a.bied-charreton@proxmox.com/
[v4] https://lore.proxmox.com/pve-devel/20261022124708.429069-1-a.bied-charreton@proxmox.com/
[v3] https://lore.proxmox.com/pve-devel/20260925094230.844917-1-a.bied-charreton@proxmox.com/
[v2] https://lore.proxmox.com/pve-devel/20260818133413.450776-1-a.bied-charreton@proxmox.com/
[v1] https://lore.proxmox.com/pve-devel/20260407073658.90818-1-a.bied-charreton@proxmox.com/
[0] https://lore.proxmox.com/pve-devel/awaoshjzbr7adisjngsrcts4zs3hxgoxw2lgcoq66gtiap3al2@mpxrbmtyfcwj/
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=8176
pve-firewall:
Arthur Bied-Charreton (15):
firewall: run make tidy
firewall: move object reference helpers to their own module
firewall: references: add helpers to update object references
firewall: references: log the references an object update touched
firewall: references: add wrappers for the SDN-generated IPSets
firewall: parser: do not log errors for disabled rules
firewall: ipset: add option to handle dangling references on rename
firewall: ipset: add option to handle dangling references on delete
firewall: aliases: add option to handle dangling references on rename
firewall: aliases: add option to handle dangling references on delete
firewall: api: run reference updates as tasks
firewall: tests: add tests for object reference updates
firewall: references: handle security groups
firewall: groups: add option to handle dangling references on rename
fix #8176: firewall: groups: handle dangling references on delete
src/PVE/API2/Firewall/Aliases.pm | 209 ++++++++---
src/PVE/API2/Firewall/Cluster.pm | 6 +-
src/PVE/API2/Firewall/Groups.pm | 182 +++++----
src/PVE/API2/Firewall/Helpers.pm | 3 +-
src/PVE/API2/Firewall/IPSet.pm | 200 +++++++---
src/PVE/API2/Firewall/Rules.pm | 49 ++-
src/PVE/API2/Firewall/VM.pm | 7 +-
src/PVE/Firewall.pm | 44 ++-
src/PVE/Firewall/Helpers.pm | 38 --
src/PVE/Firewall/Makefile | 2 +-
src/PVE/Firewall/References.pm | 613 +++++++++++++++++++++++++++++++
src/test/Makefile | 2 +-
src/test/test_update_refs.pl | 417 +++++++++++++++++++++
13 files changed, 1514 insertions(+), 258 deletions(-)
create mode 100644 src/PVE/Firewall/References.pm
create mode 100755 src/test/test_update_refs.pl
pve-network:
Arthur Bied-Charreton (1):
apply: add option to handle dangling references on VNet deletion
src/PVE/API2/Network/SDN.pm | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
qemu-server:
Arthur Bied-Charreton (1):
api: destroy_vm: add option to handle dangling IPSet references
src/PVE/API2/Qemu.pm | 14 ++++++++++++++
1 file changed, 14 insertions(+)
pve-container:
Arthur Bied-Charreton (1):
api: destroy_vm: add option to handle dangling IPSet references
src/PVE/API2/LXC.pm | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
pve-manager:
Arthur Bied-Charreton (8):
ui: firewall: add a common widget for handling dangling references
ui: firewall: ipset: add controls for dangling references
ui: firewall: aliases: add controls for dangling references
ui: sdn: apply: add control for dangling IPSet references
ui: guest destroy: use let for non-constant variable bindings
ui: guest destroy: add control for dangling IPSet references
ui: firewall: show the task for reference updates
ui: firewall: groups: add controls for dangling references
www/manager6/Makefile | 1 +
www/manager6/Utils.js | 7 ++
www/manager6/dc/SecurityGroups.js | 81 ++++++++++++-
www/manager6/grid/FirewallAliases.js | 125 ++++++++++++++++----
www/manager6/grid/FirewallObjectCommon.js | 136 ++++++++++++++++++++++
www/manager6/panel/IPSet.js | 84 +++++++++++--
www/manager6/sdn/StatusView.js | 119 +++++++++++++++----
www/manager6/window/SafeDestroyGuest.js | 70 ++++++++++-
8 files changed, 566 insertions(+), 57 deletions(-)
create mode 100644 www/manager6/grid/FirewallObjectCommon.js
pve-docs:
Arthur Bied-Charreton (2):
firewall: remove trailing spaces
firewall: document renaming and removing objects
pve-firewall.adoc | 86 +++++++++++++++++++++++++++++++++++++++++++----
pvesdn.adoc | 9 +++--
2 files changed, 86 insertions(+), 9 deletions(-)
Summary over all repositories:
26 files changed, 2221 insertions(+), 325 deletions(-)
--
Generated by murpp 0.12.1
next reply other threads:[~2026-11-12 15:12 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 15:10 Arthur Bied-Charreton [this message]
2026-11-12 15:10 ` [PATCH pve-firewall v7 01/28] firewall: run make tidy Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 02/28] firewall: move object reference helpers to their own module Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 03/28] firewall: references: add helpers to update object references Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 04/28] firewall: references: log the references an object update touched Arthur Bied-Charreton
2026-11-13 7:15 ` Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 05/28] firewall: references: add wrappers for the SDN-generated IPSets Arthur Bied-Charreton
2026-11-13 7:19 ` Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 06/28] firewall: parser: do not log errors for disabled rules Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 07/28] firewall: ipset: add option to handle dangling references on rename Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 08/28] firewall: ipset: add option to handle dangling references on delete Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 09/28] firewall: aliases: add option to handle dangling references on rename Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 10/28] firewall: aliases: add option to handle dangling references on delete Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 11/28] firewall: api: run reference updates as tasks Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 12/28] firewall: tests: add tests for object reference updates Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 13/28] firewall: references: handle security groups Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 14/28] firewall: groups: add option to handle dangling references on rename Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-firewall v7 15/28] fix #8176: firewall: groups: handle dangling references on delete Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-network v7 16/28] apply: add option to handle dangling references on VNet deletion Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH qemu-server v7 17/28] api: destroy_vm: add option to handle dangling IPSet references Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-container v7 18/28] " Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 19/28] ui: firewall: add a common widget for handling dangling references Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 20/28] ui: firewall: ipset: add controls for " Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 21/28] ui: firewall: aliases: " Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 22/28] ui: sdn: apply: add control for dangling IPSet references Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 23/28] ui: guest destroy: use let for non-constant variable bindings Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 24/28] ui: guest destroy: add control for dangling IPSet references Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 25/28] ui: firewall: show the task for reference updates Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-manager v7 26/28] ui: firewall: groups: add controls for dangling references Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-docs v7 27/28] firewall: remove trailing spaces Arthur Bied-Charreton
2026-11-12 15:10 ` [PATCH pve-docs v7 28/28] firewall: document renaming and removing objects Arthur Bied-Charreton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261112151045.577923-1-a.bied-charreton@proxmox.com \
--to=a.bied-charreton@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.