public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Dominik Csapak <d.csapak@proxmox.com>
To: Thomas Lamprecht <t.lamprecht@proxmox.com>, pve-devel@lists.proxmox.com
Subject: Re: [PATCH cluster/guest-common/qemu-server/container/manager/docs 00/34] fix #7846: guest identity (uid) and cluster ID
Date: Tue, 17 Nov 2026 11:47:15 +0100	[thread overview]
Message-ID: <5aaeb776-c738-4e0a-bc6b-814b56f24b06@proxmox.com> (raw)
In-Reply-To: <20261116004703.2908294-1-t.lamprecht@proxmox.com>

I started reviewing this series and wanted to give some early feedback 
on the overall approach. More in depth code review see the individual
patches (might take a bit going through every patch)

Note that i might have overlooked something. If so, please
do tell me.

My high level feedback:

Cluster IDs:

Is there a particular reason why 100bit + base32?
I couldn't find a rationale for the choice (did i overlook it?)
and I think using something 'industry standard' like a
UUIDv7 would be a better choice? I'ts just 28 bit more (
with a bit less randomness) but with a time stamp included and this
should make each cluster uniquely identifiable too.

If we want to stay on base32, maybe 80 or 120 bit is a better choice,
since we then don't have to use half a byte (100bit => 12.5 byte)
and still provide enough randomness.

VM UIDs:

Same question about the rationale behind the format, but with one
change: the 'registering' time is already recorded in
the meta object (as ctime), no? so providing an additional separate
timestamp seems weird to me. Also using UUIDv7 here
would include a timestamp too and also be more 'industry standard'
while only using 1 byte extra (36 vs 20 + 15). Though
we probably would save some space by not having two properties in
the meta object.

UID adoption/checks:

The cluster wide check is necessary, i get it. Handling
it with a cluster lock + reading the cofnig
could make that lookup a bottleneck in big and dynamic clusters?
Or did i misunderstood how we get the meta config out?

Alternatively we could have that lookup in pmxcfs itself and
replicate it like rrd, etc.? having a uuid saved as
a 128bit number (16byte) would mean

160k for 10000 guests
16M for 1000000 guests

which looks reasonable to me?
(we already have to have much more data in pmxcfs
with that many guests...)


Let me know what you think


On 11/16/26 1:48 AM, Thomas Lamprecht wrote:
> A VMID can be reused for another guest, and a guest can get another
> VMID or move to another cluster through a restore or a remote
> migration. So backup software and other tools cannot tell whether they
> still look at the same guest, see [0].
> 
> This series gives every VM and container a guest identity in the
> read-only 'meta' property: a random uid (100 bits as 20 base32
> characters) and the time the guest was registered in the cluster. Create
> and clone get a new uid. Config changes, migration, rollback, backup and
> restore keep it, also under another VMID or in another cluster, as long
> as no other guest there has it. Each cluster or standalone node also
> gets a cluster ID in the same format, generated and protected by pmxcfs.
> The docs patches describe the details.
> 
> Two behavior changes come with it:
> 
> * Restoring over an existing guest whose uid differs from the one of
>    the backup is refused unless 'allow-identity-mismatch' is set. 'force'
>    alone does not override this.
> * Older versions drop the whole 'meta' property of a guest that has an
>    identity. So migration and clone to such nodes is refused, and remote
>    migration bumps the tunnel version. This also hits moves by the HA
>    stack, which stops a container or a relocated VM before the migration
>    gets refused. A target check in the LRM that avoids this will follow
>    separately.
> 
> The patches are in dependency order: guest-common needs a pve-cluster
> newer than 9.1.9, qemu-server and container need guest-common, and
> manager needs all of them. manager also has to depend on the
> qemu-server and container versions with this series, as the migration
> guard takes the pve-manager version of a node as proof that the node
> keeps the identity. MIN_PVE_MANAGER_VERSION in guest-common is a
> stand-in for that pve-manager version.
> 
> Tested with the test suites of the repositories and end to end on a
> nested three-node cluster that was upgraded node by node.
> 
> [0]: https://bugzilla.proxmox.com/show_bug.cgi?id=7846
> 
> cluster:
> 
> Thomas Lamprecht (7):
>    pmxcfs: reject changes to the cluster ID file from user space
>    pmxcfs: generate a cluster ID on standalone nodes
>    pmxcfs: dfsm: add a callback that runs once all members are in sync
>    pmxcfs: quorum: pass the corosync membership to an optional listener
>    pmxcfs: dcdb: generate the cluster ID once the cluster is in sync
>    cluster: add helper to read the cluster ID
>    pmxcfs: allow removing the cluster ID file in forced local mode
> 
>   src/PVE/Cluster.pm          |  16 ++
>   src/pmxcfs/Makefile         |  12 +-
>   src/pmxcfs/cfs-plug-memdb.c |  46 ++++
>   src/pmxcfs/cfs-utils.c      |   8 +
>   src/pmxcfs/cfs-utils.h      |   5 +
>   src/pmxcfs/check_dcdb.c     | 529 ++++++++++++++++++++++++++++++++++++
>   src/pmxcfs/check_memdb.c    | 311 +++++++++++++++++++++
>   src/pmxcfs/cluster-id.c     | 158 +++++++++++
>   src/pmxcfs/cluster-id.h     |  52 ++++
>   src/pmxcfs/dcdb.c           | 144 ++++++++++
>   src/pmxcfs/dcdb.h           |   5 +
>   src/pmxcfs/dfsm.c           |  58 ++++
>   src/pmxcfs/dfsm.h           |  14 +
>   src/pmxcfs/pmxcfs.c         |  10 +-
>   src/pmxcfs/quorum.c         |  26 +-
>   src/pmxcfs/quorum.h         |   8 +-
>   src/pmxcfs/status.c         |  12 +
>   src/pmxcfs/status.h         |   3 +
>   src/test/Makefile           |   6 +-
>   src/test/test_cluster_id.pl |  47 ++++
>   20 files changed, 1458 insertions(+), 12 deletions(-)
>   create mode 100644 src/pmxcfs/check_dcdb.c
>   create mode 100644 src/pmxcfs/cluster-id.c
>   create mode 100644 src/pmxcfs/cluster-id.h
>   create mode 100644 src/test/test_cluster_id.pl
> 
> 
> guest-common:
> 
> Thomas Lamprecht (4):
>    guest identity: add helpers for the guest uid and registration time
>    abstract config: assign a guest identity and keep it on config writes
>    abstract config: let the VMID reservation return the previous state
>    guest identity: allow taking over a uid that is unused in the cluster
> 
>   debian/control                     |    1 +
>   src/Makefile                       |    1 +
>   src/PVE/AbstractConfig.pm          |  235 +++++-
>   src/PVE/GuestIdentity.pm           |  384 ++++++++++
>   src/tests/Makefile                 |   10 +-
>   src/tests/config-identity-tests.pl | 1090 ++++++++++++++++++++++++++++
>   src/tests/guest-identity-tests.pl  |  421 +++++++++++
>   7 files changed, 2135 insertions(+), 7 deletions(-)
>   create mode 100644 src/PVE/GuestIdentity.pm
>   create mode 100755 src/tests/config-identity-tests.pl
>   create mode 100755 src/tests/guest-identity-tests.pl
> 
> 
> qemu-server:
> 
> Thomas Lamprecht (6):
>    api: restore: release the VMID if a restore to an unused VMID fails
>    api: migrate, clone: refuse targets that would drop the guest identity
>    fix #7846: config: add a guest identity to the 'meta' property
>    restore: take over the guest uid from the backup where possible
>    api: remote migration: take over the guest uid from the source
>    api: status: return the guest uid
> 
>   src/PVE/API2/Qemu.pm                          | 104 +-
>   src/PVE/CLI/qmrestore.pm                      |   7 +
>   src/PVE/QemuConfig.pm                         |   7 +
>   src/PVE/QemuMigrate.pm                        |   9 +-
>   src/PVE/QemuServer.pm                         | 150 ++-
>   src/PVE/QemuServer/MetaInfo.pm                |   2 +
>   src/test/Makefile                             |   5 +-
>   src/test/QemuServerTest/FakeCFS.pm            | 197 ++++
>   .../q35-windows-pinning-identity-only.conf    |   6 +
>   ...q35-windows-pinning-identity-only.conf.cmd |  26 +
>   .../meta-identity-invalid.conf                |   6 +
>   .../meta-identity-invalid.conf.strict.error   |   2 +
>   .../meta-identity-invalid.conf                |   7 +
>   .../parse-config-input/meta-identity.conf     |  20 +
>   src/test/restore-config-expected/150.conf     |   9 +
>   src/test/restore-config-input/150.conf        |  10 +
>   src/test/run_api_tests.pl                     | 157 +++
>   src/test/run_guest_identity_tests.pl          | 533 ++++++++++
>   src/test/run_parse_config_tests.pl            |   2 +-
>   src/test/run_qemu_restore_config_tests.pl     | 916 +++++++++++++++++-
>   .../create/qemu-server/401.conf               |  31 +
>   .../create/qemu-server/402.conf               |  31 +
>   .../rollback/qemu-server/401.conf             |  36 +
>   .../rollback/qemu-server/402.conf             |  35 +
>   .../create/qemu-server/401.conf               |  13 +
>   .../create/qemu-server/402.conf               |  13 +
>   .../rollback/qemu-server/401.conf             |  35 +
>   .../rollback/qemu-server/402.conf             |  34 +
>   src/test/snapshot-test.pm                     |  85 +-
>   29 files changed, 2439 insertions(+), 49 deletions(-)
>   create mode 100644 src/test/QemuServerTest/FakeCFS.pm
>   create mode 100644 src/test/cfg2cmd/q35-windows-pinning-identity-only.conf
>   create mode 100644 src/test/cfg2cmd/q35-windows-pinning-identity-only.conf.cmd
>   create mode 100644 src/test/parse-config-expected/meta-identity-invalid.conf
>   create mode 100644 src/test/parse-config-expected/meta-identity-invalid.conf.strict.error
>   create mode 100644 src/test/parse-config-input/meta-identity-invalid.conf
>   create mode 100644 src/test/parse-config-input/meta-identity.conf
>   create mode 100644 src/test/restore-config-expected/150.conf
>   create mode 100644 src/test/restore-config-input/150.conf
>   create mode 100755 src/test/run_guest_identity_tests.pl
>   create mode 100644 src/test/snapshot-expected/create/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-expected/create/qemu-server/402.conf
>   create mode 100644 src/test/snapshot-expected/rollback/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-expected/rollback/qemu-server/402.conf
>   create mode 100644 src/test/snapshot-input/create/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-input/create/qemu-server/402.conf
>   create mode 100644 src/test/snapshot-input/rollback/qemu-server/401.conf
>   create mode 100644 src/test/snapshot-input/rollback/qemu-server/402.conf
> 
> 
> container:
> 
> Thomas Lamprecht (8):
>    config: add read-only meta property with the creation time
>    fix #7846: config: add a guest identity to the 'meta' property
>    api: migrate, clone: refuse targets that would drop the guest identity
>    restore: take over the guest uid from the backup where possible
>    restore: refuse overwriting a container with another guest's backup
>    migrate: remote: clean up the target on a tunnel version mismatch
>    api: remote migration: take over the guest uid from the source
>    api: status: return the guest uid
> 
>   src/PVE/API2/LXC.pm         |  166 +++-
>   src/PVE/API2/LXC/Config.pm  |   29 +-
>   src/PVE/API2/LXC/Status.pm  |   10 +
>   src/PVE/LXC/Config.pm       |   63 +-
>   src/PVE/LXC/Migrate.pm      |   31 +-
>   src/test/Makefile           |    8 +-
>   src/test/config_tests.pl    |   23 +
>   src/test/lifecycle_tests.pl | 1878 +++++++++++++++++++++++++++++++++++
>   8 files changed, 2177 insertions(+), 31 deletions(-)
>   create mode 100755 src/test/lifecycle_tests.pl
> 
> 
> manager:
> 
> Thomas Lamprecht (5):
>    api: cluster resources: add guest uid and registration time
>    api: cluster status: add cluster ID
>    ui: restore: allow overwriting a guest with a backup of another guest
>    ui: migrate: refuse target nodes that would drop the guest identity
>    ui: guest summary: show the guest identity
> 
>   PVE/API2/Cluster.pm                      |  35 +++-
>   test/ClusterResources_test.pl            | 249 +++++++++++++++++++++++
>   test/ClusterStatus_test.pl               | 114 +++++++++++
>   test/Makefile                            |  10 +-
>   www/manager6/Makefile                    |   1 +
>   www/manager6/panel/GuestIdentity.js      | 142 +++++++++++++
>   www/manager6/panel/GuestStatusView.js    |   3 +
>   www/manager6/panel/TemplateStatusView.js |   3 +
>   www/manager6/window/Migrate.js           |  29 +++
>   www/manager6/window/Restore.js           |  18 ++
>   10 files changed, 602 insertions(+), 2 deletions(-)
>   create mode 100755 test/ClusterResources_test.pl
>   create mode 100755 test/ClusterStatus_test.pl
>   create mode 100644 www/manager6/panel/GuestIdentity.js
> 
> 
> docs:
> 
> Thomas Lamprecht (4):
>    vzdump: describe the guest identity of VMs and containers
>    pvecm: document the cluster ID and how to get a new one
>    qm: configuration: describe the read-only meta property
>    pct: configuration: describe the read-only meta property
> 
>   pct.adoc    |  35 +++++++
>   pmxcfs.adoc |   1 +
>   pvecm.adoc  |  86 +++++++++++++++++
>   qm.adoc     |  38 ++++++++
>   vzdump.adoc | 259 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>   5 files changed, 419 insertions(+)
> 
> 
> Summary over all repositories:
>    79 files changed, 9230 insertions(+), 101 deletions(-)
> 





  parent reply	other threads:[~2026-11-17 10:47 UTC|newest]

Thread overview: 49+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16  0:46 [PATCH cluster/guest-common/qemu-server/container/manager/docs 00/34] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-16  0:46 ` [PATCH cluster 01/34] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-16  0:46 ` [PATCH cluster 02/34] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-16  0:46 ` [PATCH cluster 03/34] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-16  0:46 ` [PATCH cluster 04/34] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-16  0:46 ` [PATCH cluster 05/34] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-16  0:46 ` [PATCH cluster 06/34] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-16  0:46 ` [PATCH cluster 07/34] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-16  0:46 ` [PATCH guest-common 08/34] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-17 10:47   ` Dominik Csapak
2026-11-16  0:46 ` [PATCH guest-common 09/34] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-17 10:47   ` Dominik Csapak
2026-11-17 12:47     ` Dominik Csapak
2026-11-16  0:46 ` [PATCH guest-common 10/34] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-16  0:46 ` [PATCH guest-common 11/34] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-17 13:14   ` Dominik Csapak
2026-11-16  0:46 ` [PATCH qemu-server 12/34] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-16  0:46 ` [PATCH qemu-server 13/34] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-16  0:46 ` [PATCH qemu-server 14/34] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-16  0:46 ` [PATCH qemu-server 15/34] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-17 15:17   ` Dominik Csapak
2026-11-16  0:46 ` [PATCH qemu-server 16/34] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-17 15:26   ` Dominik Csapak
2026-11-16  0:46 ` [PATCH qemu-server 17/34] api: status: return the guest uid Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 18/34] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 19/34] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 20/34] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 21/34] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 22/34] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 23/34] migrate: remote: clean up the target on a tunnel version mismatch Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 24/34] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-16  0:46 ` [PATCH container 25/34] api: status: return the guest uid Thomas Lamprecht
2026-11-16  0:46 ` [PATCH manager 26/34] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-17 15:30   ` Dominik Csapak
2026-11-16  0:46 ` [PATCH manager 27/34] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-16  0:46 ` [PATCH manager 28/34] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-16  0:46 ` [PATCH manager 29/34] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-16  0:46 ` [PATCH manager 30/34] ui: guest summary: show " Thomas Lamprecht
2026-11-17 15:44   ` Dominik Csapak
2026-11-16  0:46 ` [PATCH docs 31/34] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-16  0:46 ` [PATCH docs 32/34] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-16  0:46 ` [PATCH docs 33/34] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-16  0:46 ` [PATCH docs 34/34] pct: " Thomas Lamprecht
2026-11-17 10:47 ` Dominik Csapak [this message]
2026-11-17 11:29   ` [PATCH cluster/guest-common/qemu-server/container/manager/docs 00/34] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-17 11:55     ` Dominik Csapak
2026-11-17 12:47       ` Thomas Lamprecht
2026-11-17 13:54         ` Dominik Csapak
2026-11-17 15:55 ` Dominik Csapak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5aaeb776-c738-4e0a-bc6b-814b56f24b06@proxmox.com \
    --to=d.csapak@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    --cc=t.lamprecht@proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal