public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Dominik Csapak <d.csapak@proxmox.com>
To: Thomas Lamprecht <t.lamprecht@proxmox.com>, pve-devel@lists.proxmox.com
Subject: Re: [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID
Date: Wed, 18 Nov 2026 14:49:27 +0100	[thread overview]
Message-ID: <5c3aa7d2-aea2-462a-80a8-bfcf95c18247@proxmox.com> (raw)
In-Reply-To: <20261118043817.1659212-1-t.lamprecht@proxmox.com>

Read through it again (though mostly via diff from the v1)
and it looks good to me now:

* the rationale is given
* the flow is better visible (and with better wording)
* remote migrate is now blocked by default on conflicting uids

the biggest issue to discuss maybe still is if we want to allow
restore with a new uid in case the same uid exists already,
same argument as for the remote migration:

if the same vm already exists, restoring it again can have
bad implications like duplicate ips, etc.

but IMHO it's not a blocker, maybe even something the admin
would want to configure globally? (datacenter.cfg?)

one thing i noticed during testing was if i modified a guest
to have an invalid uid, on the next config write, the
ctime and creation-qemu vanished from the config and
only a new rtime and uid remained.
(though i had to manually edit the config, so not a problem IMO)

there are still some nits here and there, but they are mostly
taste questions and nothing that can't be improved further down the line

for example: when restoring a vm over another another, the failure i get is:

TASK ERROR: command 'set -o pipefail && zstd -q -d -c -- 
/var/lib/vz/dump/vzdump-qemu-9999-2026_11_18-14_37_58.vma.zst | vma 
extract -v -r /var/tmp/vzdumptmp8272.fifo - /var/tmp/vzdumptmp8272' 
failed: backup belongs to another guest: its uid '2xuof4nw6rnowo5mpgs2' 
differs from uid 'itomggi4ofm33szeb3h2' of the existing VM 102 - use 
'allow-identity-mismatch' to overwrite the VM anyway

which is not very readable in the task log window  (still functional though)

also at least a few things in the ui i'm missing:

a way to show the cluster/node id (probably in the datacenter summary)
using the uid (maybe rtime) as variables in the templating for the notes

Tested it on a test cluster with some backup/restore cycles,
adopting of uid, manually editing configs, etc.

Since nothing of these blocks this series from my POV, consider it:

Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>


On 11/18/26 5:39 AM, Thomas Lamprecht wrote:
> A VMID can be reused for another guest, and a guest can get another
> VMID or move to another cluster through a restore or a remote
> migration. So backup software and other tools cannot tell whether they
> still look at the same guest, see [0].
> 
> This series gives every VM and container a guest identity in the
> read-only 'meta' property: a random uid (100 bits as 20 base32
> characters) and the time the guest was registered in the cluster. Create
> and clone get a new uid. Config changes, migration, rollback, backup and
> restore keep it, also under another VMID or in another cluster, as long
> as no other guest there has it. Each cluster or standalone node also
> gets a cluster ID in the same format, generated and protected by pmxcfs.
> The docs patches describe the details.
> 
> The uid is a plain random nonce: 100 bits make a duplicate practically
> impossible and fit into a 128-bit value with room to spare, and 20
> base32 characters stay short and need no quoting. The registration time
> is separate from the creation time, which clones, restores and remote
> migrations copy along. It changes when a guest gets registered again, so
> tools can tell such a guest from the one they knew.
> 
> Three behavior changes come with it:
> 
> * Restoring over an existing guest whose uid differs from the one of
>    the backup is refused unless 'allow-identity-mismatch' is set. 'force'
>    alone does not override this.
> * A migration to another cluster is refused if another guest there
>    already has the uid, unless 'allow-new-uid' is set. The check runs
>    before anything is transferred. PDM does not know the option yet.
> * Older versions drop the whole 'meta' property of a guest that has an
>    identity. So migration and clone to such nodes is refused, and remote
>    migration bumps the tunnel version. This also hits moves by the HA
>    stack, which stops a container or a relocated VM before the migration
>    gets refused. A target check in the LRM that avoids this will follow
>    separately.
> 
> The patches are in dependency order: guest-common needs the pve-cluster
> of this series, qemu-server and container need guest-common, and manager
> needs all of them. manager also has to depend on the qemu-server and
> container versions with this series, as the migration guard takes the
> pve-manager version of a node as proof that the node keeps the identity.
> MIN_PVE_MANAGER_VERSION in guest-common is a stand-in for that version.
> 
> Tested with the affected tests of the repositories for every patch and
> their full suites at the end, pmxcfs with corosync in network
> namespaces, and end to end on a nested three-node cluster that was
> upgraded node by node, with remote migrations to a standalone node.
> 
> changes v2 -> v3:
> * remote migration: a guest without uid gets one before the source
>    sends it, so source and target no longer end up with unrelated uids;
>    the identity command moved to pve-guest-common for both guest types
>    (new patch 13)
> * restore: keep the guest if a restored volume or firewall config
>    cannot be removed, fail if a container backup changes between its
>    check and its extraction, and let the cleanup follow the reservation
> * reserving a VMID refuses an existing config that fails to load
>    instead of replacing it
> * cluster ID: generate it right away when the configured nodes change
>    (new patch 5)
> * feature API without version lookups for snapshot requests, and the
>    cluster-wide bulk migration refuses its target up front
> * docs: failed restores, uid conflicts on migration, qm.conf(5) and
>    pct.conf(5)
> * more independent fixes that the series builds on were applied on
>    their own: failed and concurrent restores and creates, the schema of
>    the migration preconditions, the lock of a VMA restore, a node list
>    cut short in pmxcfs
> 
> changes v1 -> v2, thanks to Dominik for the review:
> * rebase onto current master (tracking of used guest IDs, dfsm rework)
> * remote migration takes the uid over at its start and refuses a uid
>    that is in use in the target cluster, unless 'allow-new-uid' is set
> * qemu-server and container share the restore checks and messages
>    through pve-guest-common, with fewer special cases
> * IDs take each character from one random byte, without an encoder
> * the clone dialog does not offer nodes that would drop the identity,
>    and a bulk migration checks its target once (patch 34)
> * independent fixes moved out of the series and got applied on their
>    own: the IPC client error, the order of removing firewall and guest
>    config, and the cleanup of a remote migration target after failures
> * comments, commit messages and docs give the reasons for the format,
>    for rtime next to ctime and for the separate lock
> Details are in the notes below the "---" of each patch.
> 
> v2: https://lore.proxmox.com/pve-devel/20261117223945.862914-1-t.lamprecht@proxmox.com/
> v1: https://lore.proxmox.com/pve-devel/20261116004703.2908294-1-t.lamprecht@proxmox.com/
> 
> [0]: https://bugzilla.proxmox.com/show_bug.cgi?id=7846
> 
> cluster:
> 
> Thomas Lamprecht (8):
>    pmxcfs: reject changes to the cluster ID file from user space
>    pmxcfs: generate a cluster ID on standalone nodes
>    pmxcfs: dfsm: add a callback that runs once all members are in sync
>    pmxcfs: quorum: pass the corosync membership to an optional listener
>    pmxcfs: confdb: call an optional listener when the nodes get published
>    pmxcfs: dcdb: generate the cluster ID once the cluster is in sync
>    cluster: add helper to read the cluster ID
>    pmxcfs: allow removing the cluster ID file in forced local mode
> 
>   src/PVE/Cluster.pm          |  16 +
>   src/pmxcfs/Makefile         |  12 +-
>   src/pmxcfs/cfs-plug-memdb.c |  46 +++
>   src/pmxcfs/cfs-utils.c      |   8 +
>   src/pmxcfs/cfs-utils.h      |   5 +
>   src/pmxcfs/check_dcdb.c     | 563 ++++++++++++++++++++++++++++++++++++
>   src/pmxcfs/check_memdb.c    | 283 ++++++++++++++++++
>   src/pmxcfs/cluster-id.c     | 142 +++++++++
>   src/pmxcfs/cluster-id.h     |  50 ++++
>   src/pmxcfs/confdb.c         |  18 +-
>   src/pmxcfs/confdb.h         |   5 +-
>   src/pmxcfs/dcdb.c           | 153 ++++++++++
>   src/pmxcfs/dcdb.h           |   8 +
>   src/pmxcfs/dfsm.c           |  59 ++++
>   src/pmxcfs/dfsm.h           |  14 +
>   src/pmxcfs/pmxcfs.c         |  12 +-
>   src/pmxcfs/quorum.c         |  26 +-
>   src/pmxcfs/quorum.h         |   9 +-
>   src/pmxcfs/status.c         |  12 +
>   src/pmxcfs/status.h         |   3 +
>   src/test/Makefile           |   6 +-
>   src/test/test_cluster_id.pl |  46 +++
>   22 files changed, 1478 insertions(+), 18 deletions(-)
>   create mode 100644 src/pmxcfs/check_dcdb.c
>   create mode 100644 src/pmxcfs/cluster-id.c
>   create mode 100644 src/pmxcfs/cluster-id.h
>   create mode 100644 src/test/test_cluster_id.pl
> 
> 
> guest-common:
> 
> Thomas Lamprecht (5):
>    guest identity: add helpers for the guest uid and registration time
>    abstract config: assign a guest identity and keep it on config writes
>    abstract config: let the VMID reservation return the previous state
>    guest identity: allow taking over a uid that is unused in the cluster
>    migrate: add shared helpers for the identity command of the migration tunnel
> 
>   src/Makefile                       |    1 +
>   src/PVE/AbstractConfig.pm          |  202 +++-
>   src/PVE/AbstractMigrate.pm         |   30 +
>   src/PVE/GuestIdentity.pm           |  510 ++++++++++
>   src/tests/Makefile                 |   10 +-
>   src/tests/config-identity-tests.pl | 1391 ++++++++++++++++++++++++++++
>   src/tests/guest-identity-tests.pl  |  530 +++++++++++
>   7 files changed, 2668 insertions(+), 6 deletions(-)
>   create mode 100644 src/PVE/GuestIdentity.pm
>   create mode 100755 src/tests/config-identity-tests.pl
>   create mode 100755 src/tests/guest-identity-tests.pl
> 
> 
> qemu-server:
> 
> Thomas Lamprecht (7):
>    api: restore: release the VMID if a restore to an unused VMID fails
>    api: migrate, clone: refuse targets that would drop the guest identity
>    fix #7846: config: add a guest identity to the 'meta' property
>    restore: take over the guest uid from the backup where possible
>    api: remote migration: take over the guest uid from the source
>    api: status: return the guest uid
>    tests: remote migration: check a failed command and an early tunnel end
> 
>   src/PVE/API2/Qemu.pm                          |  118 +-
>   src/PVE/CLI/qm.pm                             |    7 +
>   src/PVE/CLI/qmrestore.pm                      |   11 +-
>   src/PVE/QemuConfig.pm                         |    7 +
>   src/PVE/QemuMigrate.pm                        |   18 +-
>   src/PVE/QemuServer.pm                         |  155 ++-
>   src/PVE/QemuServer/MetaInfo.pm                |    2 +
>   src/test/Makefile                             |    5 +-
>   src/test/QemuServerTest/FakeCFS.pm            |  189 +++
>   .../q35-windows-pinning-identity-only.conf    |    6 +
>   ...q35-windows-pinning-identity-only.conf.cmd |   26 +
>   .../meta-identity-invalid.conf                |    6 +
>   .../meta-identity-invalid.conf.strict.error   |    2 +
>   .../meta-identity-invalid.conf                |    7 +
>   .../parse-config-input/meta-identity.conf     |   20 +
>   src/test/restore-config-expected/150.conf     |    9 +
>   src/test/restore-config-input/150.conf        |   10 +
>   src/test/run_api_tests.pl                     |  206 ++++
>   src/test/run_guest_identity_tests.pl          |  803 +++++++++++++
>   src/test/run_parse_config_tests.pl            |    2 +-
>   src/test/run_qemu_restore_config_tests.pl     | 1053 ++++++++++++++++-
>   .../create/qemu-server/401.conf               |   31 +
>   .../create/qemu-server/402.conf               |   31 +
>   .../rollback/qemu-server/401.conf             |   36 +
>   .../rollback/qemu-server/402.conf             |   35 +
>   .../create/qemu-server/401.conf               |   13 +
>   .../create/qemu-server/402.conf               |   13 +
>   .../rollback/qemu-server/401.conf             |   35 +
>   .../rollback/qemu-server/402.conf             |   34 +
>   src/test/snapshot-test.pm                     |   82 +-
>   30 files changed, 2899 insertions(+), 73 deletions(-)
>   create mode 100644 src/test/QemuServerTest/FakeCFS.pm
>   create mode 100644 src/test/cfg2cmd/q35-windows-pinning-identity-only.conf
>   create mode 100644 src/test/cfg2cmd/q35-windows-pinning-identity-only.conf.cmd
>   create mode 100644 src/test/parse-config-expected/meta-identity-invalid.conf
>   create mode 100644 src/test/parse-config-expected/meta-identity-invalid.conf.strict.error
>   create mode 100644 src/test/parse-config-input/meta-identity-invalid.conf
>   create mode 100644 src/test/parse-config-input/meta-identity.conf
>   create mode 100644 src/test/restore-config-expected/150.conf
>   create mode 100644 src/test/restore-config-input/150.conf
>   create mode 100755 src/test/run_guest_identity_tests.pl
>   create mode 100644 src/test/snapshot-expected/create/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-expected/create/qemu-server/402.conf
>   create mode 100644 src/test/snapshot-expected/rollback/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-expected/rollback/qemu-server/402.conf
>   create mode 100644 src/test/snapshot-input/create/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-input/create/qemu-server/402.conf
>   create mode 100644 src/test/snapshot-input/rollback/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-input/rollback/qemu-server/402.conf
> 
> 
> container:
> 
> Thomas Lamprecht (8):
>    config: add read-only meta property with the creation time
>    fix #7846: config: add a guest identity to the 'meta' property
>    api: migrate, clone: refuse targets that would drop the guest identity
>    restore: take over the guest uid from the backup where possible
>    restore: refuse overwriting a container with another guest's backup
>    test: remote migration: check the cleanup of the target after failures
>    api: remote migration: take over the guest uid from the source
>    api: status: return the guest uid
> 
>   src/PVE/API2/LXC.pm         |  190 ++-
>   src/PVE/API2/LXC/Config.pm  |   29 +-
>   src/PVE/API2/LXC/Status.pm  |    6 +
>   src/PVE/CLI/pct.pm          |    7 +
>   src/PVE/LXC/Config.pm       |   51 +-
>   src/PVE/LXC/Migrate.pm      |    8 +-
>   src/test/Makefile           |    8 +-
>   src/test/config_tests.pl    |   23 +
>   src/test/lifecycle_tests.pl | 2446 +++++++++++++++++++++++++++++++++++
>   9 files changed, 2730 insertions(+), 38 deletions(-)
>   create mode 100755 src/test/lifecycle_tests.pl
> 
> 
> manager:
> 
> Thomas Lamprecht (6):
>    api: cluster resources: add guest uid and registration time
>    api: cluster status: add cluster ID
>    ui: restore: allow overwriting a guest with a backup of another guest
>    ui: migrate: refuse target nodes that would drop the guest identity
>    ui: guest summary: show the guest identity
>    api: bulk migrate: refuse targets that would drop the guest identity
> 
>   PVE/API2/Cluster.pm                      |  24 ++-
>   PVE/API2/Cluster/BulkAction/Guest.pm     |   5 +
>   PVE/API2/Nodes.pm                        |   3 +
>   test/ClusterResources_test.pl            | 249 +++++++++++++++++++++++
>   test/ClusterStatus_test.pl               | 114 +++++++++++
>   test/Makefile                            |  10 +-
>   www/manager6/Makefile                    |   1 +
>   www/manager6/panel/GuestIdentity.js      | 142 +++++++++++++
>   www/manager6/panel/GuestStatusView.js    |   3 +
>   www/manager6/panel/TemplateStatusView.js |   3 +
>   www/manager6/window/Migrate.js           |  29 +++
>   www/manager6/window/Restore.js           |  18 ++
>   12 files changed, 599 insertions(+), 2 deletions(-)
>   create mode 100755 test/ClusterResources_test.pl
>   create mode 100755 test/ClusterStatus_test.pl
>   create mode 100644 www/manager6/panel/GuestIdentity.js
> 
> 
> docs:
> 
> Thomas Lamprecht (4):
>    vzdump: describe the guest identity of VMs and containers
>    pvecm: document the cluster ID and how to get a new one
>    qm: configuration: describe the read-only meta property
>    pct: configuration: describe the read-only meta property
> 
>   pct.adoc      |  35 ++++++
>   pct.conf.adoc |   3 +
>   pmxcfs.adoc   |   1 +
>   pvecm.adoc    |  87 +++++++++++++
>   qm.adoc       |  38 ++++++
>   qm.conf.adoc  |   3 +
>   vzdump.adoc   | 335 ++++++++++++++++++++++++++++++++++++++++++++++++++
>   7 files changed, 502 insertions(+)
> 
> 
> Summary over all repositories:
>    87 files changed, 10876 insertions(+), 137 deletions(-)
> 





  parent reply	other threads:[~2026-11-18 13:49 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  4:37 [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 01/38] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 02/38] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 03/38] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 04/38] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 05/38] pmxcfs: confdb: call an optional listener when the nodes get published Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 06/38] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 07/38] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 08/38] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 09/38] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 10/38] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 11/38] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 12/38] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 13/38] migrate: add shared helpers for the identity command of the migration tunnel Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 14/38] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 15/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 16/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 17/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 18/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 19/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 20/38] tests: remote migration: check a failed command and an early tunnel end Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 21/38] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 22/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 23/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 24/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 25/38] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 26/38] test: remote migration: check the cleanup of the target after failures Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 27/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 28/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 29/38] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 30/38] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 31/38] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 32/38] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 33/38] ui: guest summary: show " Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 34/38] api: bulk migrate: refuse targets that would drop " Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 35/38] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 36/38] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 37/38] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 38/38] pct: " Thomas Lamprecht
2026-11-18 13:49 ` Dominik Csapak [this message]
2026-11-18 14:13   ` [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-19 10:38 ` superseded: " Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5c3aa7d2-aea2-462a-80a8-bfcf95c18247@proxmox.com \
    --to=d.csapak@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    --cc=t.lamprecht@proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal