From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id D42831FF0AA for ; Tue, 17 Nov 2026 12:55:56 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 09708215E8; Tue, 17 Nov 2026 12:55:52 +0100 (CET) Message-ID: Date: Tue, 17 Nov 2026 12:55:45 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH cluster/guest-common/qemu-server/container/manager/docs 00/34] fix #7846: guest identity (uid) and cluster ID To: Thomas Lamprecht , pve-devel@lists.proxmox.com References: <20261116004703.2908294-1-t.lamprecht@proxmox.com> <5aaeb776-c738-4e0a-bc6b-814b56f24b06@proxmox.com> <3f6c702a-ba0f-4e1f-ae4a-132d9b6d928a@proxmox.com> Content-Language: en-US From: Dominik Csapak In-Reply-To: <3f6c702a-ba0f-4e1f-ae4a-132d9b6d928a@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794916545523 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.141 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WJ7IIME4JXZN2UGY5NJVSLPTEAYPSG5J X-Message-ID-Hash: WJ7IIME4JXZN2UGY5NJVSLPTEAYPSG5J X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 11/17/26 12:29 PM, Thomas Lamprecht wrote: > Am 17.11.26 um 11:47 schrieb Dominik Csapak: >> I started reviewing this series and wanted to give some early feedback on the >> overall approach. More in depth code review see the individual patches (might >> take a bit going through every patch) > > thank you for taking a look, appreciated! > >> Note that i might have overlooked something. If so, please >> do tell me. >> >> My high level feedback: >> >> Cluster IDs: >> >> Is there a particular reason why 100bit + base32? > > Yes, they are much shorter serialized (20 chars) and have enough entropy > to make them more than good enough with head room while not getting > larger than needed. > >> I couldn't find a rationale for the choice (did i overlook it?) > > No, v1 only states the format (module comment in PVE::GuestIdentity, > commit messages, docs), not really the reasoning/rational behind it. > I'll expand on that for a v2 (or on applying) and will add a few > sentences to the cover letter too, as it certainly is something one can > wonder about. > >> and I think using something 'industry standard' like a >> UUIDv7 would be a better choice? I'ts just 28 bit more ( >> with a bit less randomness) but with a time stamp included and this >> should make each cluster uniquely identifiable too. > > While I thought earlier about using snowflake or some other timestamp > influenced uid, they are not ideal here as they do not relate to > anything over the lifetime of a guest w.r.t. restores and such. > >> If we want to stay on base32, maybe 80 or 120 bit is a better choice, > > I had it at 80 bits, but that is borderline w.r.t. hundreds of billions > of entities and the birthday problem, as we're getting near actual digit > percentages then (1 % chance of a clash at ~150 billion IDs). 100 bits > takes that boundary between practically impossible and a real chance of > clash in the hundreds of trillion range (1 % at ~160 trillion), which > seemed good enough for me. but this is about the cluster id, surely there aren't billions of clusters/nodes ? they also don't have to be globally unique in practice, just for one provider/customer/etc. > >> since we then don't have to use half a byte (100bit => 12.5 byte) >> and still provide enough randomness. > > Base32 has 5 bit per character, so 100 bits are always 20 chars without > padding, that I found nice. yes i meant the 'raw size' and basically always discarding half a byte of randomness, but sure. > >> VM UIDs: >> >> Same question about the rationale behind the format, but with one >> change: the 'registering' time is already recorded in >> the meta object (as ctime), no? so providing an additional separate > > ctime is creation time, which is the first creation ever. Registration > time is something else, it's the time the guest got its identity in this > cluster. > > E.g. ctime will always be kept as is for a restore to a new VMID. The > uid changes if unique is set or if another guest in the cluster > already has it (e.g. the original still exists), but otherwise stays > the same. The registration time is regenerated in any case – making > the combination of rtime and uid (and cluster ID) unique in practice. > This comes from the model that backup providers use and even if a bit > odd at first it works well for all cases (if this series here > implements it right that is...). > > That then allows one to tell apart e.g.: > - Same uid, changed rtime: the guest was registered again, for example > destroyed and restored, or moved in from another cluster. A tool can > then tell this from an "untouched" guest. but this could also be detectable via ctime (except if we don't want to touch that on restore/migration ofc) > - Same uid in two clusters, which can happen after a remote migration > that keeps the source, or a restore elsewhere. The uid together with > the cluster ID and rtime still tells the instances apart. for this the rtime is not needed though? uid + clusterid should be enough? (since same uid in one cluster can't happen) > > I tried hard to do all that's needed to cover all bases but only really > that what's needed here. Ideally we do not touch this ever again from > a principles POV (knocks wood). > but fair, having a separate rtime can make sense i guess. and with that having a time encoded in the 'uuid' makes less sense, got it. >> timestamp seems weird to me. Also using UUIDv7 here >> would include a timestamp too and also be more 'industry standard' >> while only using 1 byte extra (36 vs 20 + 15). Though > no v7 for these things for the same reason, it's rather confusing > then. The timestamp of a v7 is fixed when the ID gets generated, while > here the uid must stay the same over restores and remote migrations > and the registration time must change then, so the two cannot be one > value. uid *must* be random, that's a core property and ensures it's > never taken for anything that it really isn't. > > FWIW, my format allows to still map all uids into standard uuidv8, > so we're not locked in, but I do not want to use them and their > various versions here, simple random nonce that is formatted such > to be easy to handle (just alphanumerical) and still reasonably short. > That's my core requirement here. I.e., yes uuid might be wide spread, > but there to many tried to hard to add additional meaning to what should > be just a simple random nonce like this use case here. makes sense, though i think having 20 vs 36 bytes in a config that is already ~500 bytes by default wouldn't make much difference IMO. (also we already have at least 2 uuids in the default config, so there would be precedent to using that format) we could also already opt to use a custom uuid format with the rtime encoded instead of having a separate property, but that is probably more pain that having it separate... > >> we probably would save some space by not having two properties in >> the meta object. > > see above, required. > >> UID adoption/checks: >> >> The cluster wide check is necessary, i get it. Handling > > It's only needed when a guest takes over an existing uid, so once at > the end of a restore or of an incoming remote migration. Without it, > restoring the same backup to two VMIDs, or to a new VMID while the > original still exists, would give two guests with the same uid in one > cluster. Newly generated uids (create, clone, first write of an > existing guest after the upgrade) take no cluster lock and do no > lookup, the randomness is enough there. it's also necessary on clone and creation though? otherwise we can't be sure the random value is not in use? (unlikely but still) if you argue about billions of clusters/nodes for 100bit randomness, then i think arguing with a much higher number of vms is fair? > >> it with a cluster lock + reading the cofnig >> could make that lookup a bottleneck in big and dynamic clusters? >> Or did i misunderstood how we get the meta config out? > > The lookup does not read config files one by one, it's a single IPC > call to pmxcfs that collects the 'meta' property from its in-memory > configs, i.e. the same call /cluster/resources already does on every > request for lock and tags. The lock is held for that call plus one > config write. So I don't expect this to become a bottleneck, and once > we have pmxcfs in rust with types for such guest configs we can do > this there on a nice abstraction level while still being really fast. ok sorry must have missed that or not yet read > >> Alternatively we could have that lookup in pmxcfs itself and >> replicate it like rrd, etc.? having a uuid saved as >> a 128bit number (16byte) would mean > > no, this must IMO not be a sidecar or saved out of band, coupling it > directly to the guest config, the core entity that makes out a guest, > is IMO crucial to get a safe design. The lookup already happens in > pmxcfs as described above, just without a second copy of the data > that could get out of sync with the configs. > >> 160k for 10000 guests >> 16M for 1000000 guests >> >> which looks reasonable to me? >> (we already have to have much more data in pmxcfs >> with that many guests...) > > Probably not needed, but we can optimize the basic approach here if > needed (special IPC call for check+lock), but that's basically an > implementation detail that won't change the overall design IMO. if the overall reading isn't happening in perl i don't think we need anything specialized...