From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: Dominik Csapak <d.csapak@proxmox.com>, pve-devel@lists.proxmox.com
Subject: Re: [PATCH cluster/guest-common/qemu-server/container/manager/docs 00/34] fix #7846: guest identity (uid) and cluster ID
Date: Tue, 17 Nov 2026 12:29:04 +0100 [thread overview]
Message-ID: <3f6c702a-ba0f-4e1f-ae4a-132d9b6d928a@proxmox.com> (raw)
In-Reply-To: <5aaeb776-c738-4e0a-bc6b-814b56f24b06@proxmox.com>
Am 17.11.26 um 11:47 schrieb Dominik Csapak:
> I started reviewing this series and wanted to give some early feedback on the
> overall approach. More in depth code review see the individual patches (might
> take a bit going through every patch)
thank you for taking a look, appreciated!
> Note that i might have overlooked something. If so, please
> do tell me.
>
> My high level feedback:
>
> Cluster IDs:
>
> Is there a particular reason why 100bit + base32?
Yes, they are much shorter serialized (20 chars) and have enough entropy
to make them more than good enough with head room while not getting
larger than needed.
> I couldn't find a rationale for the choice (did i overlook it?)
No, v1 only states the format (module comment in PVE::GuestIdentity,
commit messages, docs), not really the reasoning/rational behind it.
I'll expand on that for a v2 (or on applying) and will add a few
sentences to the cover letter too, as it certainly is something one can
wonder about.
> and I think using something 'industry standard' like a
> UUIDv7 would be a better choice? I'ts just 28 bit more (
> with a bit less randomness) but with a time stamp included and this
> should make each cluster uniquely identifiable too.
While I thought earlier about using snowflake or some other timestamp
influenced uid, they are not ideal here as they do not relate to
anything over the lifetime of a guest w.r.t. restores and such.
> If we want to stay on base32, maybe 80 or 120 bit is a better choice,
I had it at 80 bits, but that is borderline w.r.t. hundreds of billions
of entities and the birthday problem, as we're getting near actual digit
percentages then (1 % chance of a clash at ~150 billion IDs). 100 bits
takes that boundary between practically impossible and a real chance of
clash in the hundreds of trillion range (1 % at ~160 trillion), which
seemed good enough for me.
> since we then don't have to use half a byte (100bit => 12.5 byte)
> and still provide enough randomness.
Base32 has 5 bit per character, so 100 bits are always 20 chars without
padding, that I found nice.
> VM UIDs:
>
> Same question about the rationale behind the format, but with one
> change: the 'registering' time is already recorded in
> the meta object (as ctime), no? so providing an additional separate
ctime is creation time, which is the first creation ever. Registration
time is something else, it's the time the guest got its identity in this
cluster.
E.g. ctime will always be kept as is for a restore to a new VMID. The
uid changes if unique is set or if another guest in the cluster
already has it (e.g. the original still exists), but otherwise stays
the same. The registration time is regenerated in any case – making
the combination of rtime and uid (and cluster ID) unique in practice.
This comes from the model that backup providers use and even if a bit
odd at first it works well for all cases (if this series here
implements it right that is...).
That then allows one to tell apart e.g.:
- Same uid, changed rtime: the guest was registered again, for example
destroyed and restored, or moved in from another cluster. A tool can
then tell this from an "untouched" guest.
- Same uid in two clusters, which can happen after a remote migration
that keeps the source, or a restore elsewhere. The uid together with
the cluster ID and rtime still tells the instances apart.
I tried hard to do all that's needed to cover all bases but only really
that what's needed here. Ideally we do not touch this ever again from
a principles POV (knocks wood).
> timestamp seems weird to me. Also using UUIDv7 here
> would include a timestamp too and also be more 'industry standard'
> while only using 1 byte extra (36 vs 20 + 15). Though
no v7 for these things for the same reason, it's rather confusing
then. The timestamp of a v7 is fixed when the ID gets generated, while
here the uid must stay the same over restores and remote migrations
and the registration time must change then, so the two cannot be one
value. uid *must* be random, that's a core property and ensures it's
never taken for anything that it really isn't.
FWIW, my format allows to still map all uids into standard uuidv8,
so we're not locked in, but I do not want to use them and their
various versions here, simple random nonce that is formatted such
to be easy to handle (just alphanumerical) and still reasonably short.
That's my core requirement here. I.e., yes uuid might be wide spread,
but there to many tried to hard to add additional meaning to what should
be just a simple random nonce like this use case here.
> we probably would save some space by not having two properties in
> the meta object.
see above, required.
> UID adoption/checks:
>
> The cluster wide check is necessary, i get it. Handling
It's only needed when a guest takes over an existing uid, so once at
the end of a restore or of an incoming remote migration. Without it,
restoring the same backup to two VMIDs, or to a new VMID while the
original still exists, would give two guests with the same uid in one
cluster. Newly generated uids (create, clone, first write of an
existing guest after the upgrade) take no cluster lock and do no
lookup, the randomness is enough there.
> it with a cluster lock + reading the cofnig
> could make that lookup a bottleneck in big and dynamic clusters?
> Or did i misunderstood how we get the meta config out?
The lookup does not read config files one by one, it's a single IPC
call to pmxcfs that collects the 'meta' property from its in-memory
configs, i.e. the same call /cluster/resources already does on every
request for lock and tags. The lock is held for that call plus one
config write. So I don't expect this to become a bottleneck, and once
we have pmxcfs in rust with types for such guest configs we can do
this there on a nice abstraction level while still being really fast.
> Alternatively we could have that lookup in pmxcfs itself and
> replicate it like rrd, etc.? having a uuid saved as
> a 128bit number (16byte) would mean
no, this must IMO not be a sidecar or saved out of band, coupling it
directly to the guest config, the core entity that makes out a guest,
is IMO crucial to get a safe design. The lookup already happens in
pmxcfs as described above, just without a second copy of the data
that could get out of sync with the configs.
> 160k for 10000 guests
> 16M for 1000000 guests
>
> which looks reasonable to me?
> (we already have to have much more data in pmxcfs
> with that many guests...)
Probably not needed, but we can optimize the basic approach here if
needed (special IPC call for check+lock), but that's basically an
implementation detail that won't change the overall design IMO.
next prev parent reply other threads:[~2026-11-17 11:29 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 0:46 [PATCH cluster/guest-common/qemu-server/container/manager/docs 00/34] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-16 0:46 ` [PATCH cluster 01/34] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-16 0:46 ` [PATCH cluster 02/34] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-16 0:46 ` [PATCH cluster 03/34] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-16 0:46 ` [PATCH cluster 04/34] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-16 0:46 ` [PATCH cluster 05/34] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-16 0:46 ` [PATCH cluster 06/34] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-16 0:46 ` [PATCH cluster 07/34] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-16 0:46 ` [PATCH guest-common 08/34] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-17 10:47 ` Dominik Csapak
2026-11-16 0:46 ` [PATCH guest-common 09/34] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-17 10:47 ` Dominik Csapak
2026-11-17 12:47 ` Dominik Csapak
2026-11-16 0:46 ` [PATCH guest-common 10/34] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-16 0:46 ` [PATCH guest-common 11/34] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-17 13:14 ` Dominik Csapak
2026-11-16 0:46 ` [PATCH qemu-server 12/34] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-16 0:46 ` [PATCH qemu-server 13/34] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-16 0:46 ` [PATCH qemu-server 14/34] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-16 0:46 ` [PATCH qemu-server 15/34] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-17 15:17 ` Dominik Csapak
2026-11-16 0:46 ` [PATCH qemu-server 16/34] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-17 15:26 ` Dominik Csapak
2026-11-16 0:46 ` [PATCH qemu-server 17/34] api: status: return the guest uid Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 18/34] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 19/34] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 20/34] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 21/34] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 22/34] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 23/34] migrate: remote: clean up the target on a tunnel version mismatch Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 24/34] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-16 0:46 ` [PATCH container 25/34] api: status: return the guest uid Thomas Lamprecht
2026-11-16 0:46 ` [PATCH manager 26/34] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-17 15:30 ` Dominik Csapak
2026-11-16 0:46 ` [PATCH manager 27/34] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-16 0:46 ` [PATCH manager 28/34] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-16 0:46 ` [PATCH manager 29/34] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-16 0:46 ` [PATCH manager 30/34] ui: guest summary: show " Thomas Lamprecht
2026-11-17 15:44 ` Dominik Csapak
2026-11-16 0:46 ` [PATCH docs 31/34] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-16 0:46 ` [PATCH docs 32/34] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-16 0:46 ` [PATCH docs 33/34] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-16 0:46 ` [PATCH docs 34/34] pct: " Thomas Lamprecht
2026-11-17 10:47 ` [PATCH cluster/guest-common/qemu-server/container/manager/docs 00/34] fix #7846: guest identity (uid) and cluster ID Dominik Csapak
2026-11-17 11:29 ` Thomas Lamprecht [this message]
2026-11-17 11:55 ` Dominik Csapak
2026-11-17 12:47 ` Thomas Lamprecht
2026-11-17 13:54 ` Dominik Csapak
2026-11-17 15:55 ` Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3f6c702a-ba0f-4e1f-ae4a-132d9b6d928a@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=d.csapak@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox