From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id E37FE1FF0E1 for ; Mon, 16 Nov 2026 01:52:06 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 3B4C221A95; Mon, 16 Nov 2026 01:48:46 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH docs 31/34] vzdump: describe the guest identity of VMs and containers Date: Mon, 16 Nov 2026 01:46:43 +0100 Message-ID: <20261116004703.2908294-32-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261116004703.2908294-1-t.lamprecht@proxmox.com> References: <20261116004703.2908294-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794790030662 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.336 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 2LN7ZXAX7PUVL664NQP63R4XRVOPG3A4 X-Message-ID-Hash: 2LN7ZXAX7PUVL664NQP63R4XRVOPG3A4 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Since Proxmox VE 9.3, VMs and containers have a guest identity in 'meta': a random uid and a registration time. Backup software and other tools can distinguish a guest from another that reuses its VMID [0]. Administrators need to understand why restores over a different guest are refused, when a restore keeps the uid of the backup, and why migrations and clones to older nodes are blocked. Those nodes drop 'meta', including VM creation information used for hardware defaults. Document the lifecycle, failed-restore retries, and mixed-version restrictions in the backup and restore chapter, where users are most likely to encounter the identity. Include migration and HA behavior there so the rules can be read together. [0]: https://bugzilla.proxmox.com/show_bug.cgi?id=7846 Signed-off-by: Thomas Lamprecht --- vzdump.adoc | 256 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 256 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 44db027..87136af 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -464,6 +464,13 @@ following CLI tools: For details see the corresponding manual pages. +When you restore a backup over an existing guest, {pve} compares the guest +identities of the backup and of the guest. If they differ, the restore is +refused, unless you set the `allow-identity-mismatch` option. The existing guest +then keeps its identity. See +xref:guest_identity_restore[Restore Over an Existing Guest] for details and +exceptions. + Bandwidth Limit ~~~~~~~~~~~~~~~ @@ -563,6 +570,255 @@ file in `/var/log/proxmox-backup/file-restore/`. The log file might contain additional information in case an attempt to restore individual files or accessing file systems contained in a backup archive fails. +[[guest_identity]] +Guest Identity +-------------- + +Since {pve} 9.3, each virtual machine and container has a guest identity: a +random identifier, `uid`, and a registration time, `rtime`. Unlike a VMID, +which can be reused after a guest is destroyed, the `uid` lets backup software +and other tools recognize the same logical guest across restores and +migrations, even under a different VMID. Whether an operation keeps the `uid` +depends on the target and the options of the operation, as described in +xref:guest_identity_lifecycle[Lifecycle]. + +[[guest_identity_values]] +Identity Values +~~~~~~~~~~~~~~~ + +The guest identity is part of the `meta` property in the guest configuration. +This property also contains the creation time `ctime` and, for VMs, the QEMU +version that the VM was created with, `creation-qemu`: + +---- +meta: creation-qemu=10.1.2,ctime=1772444467,rtime=1794320571482,uid=w5qczgiem26qnwzwsoil +---- + +`uid`:: + +A random identifier of 20 characters, consisting of the lowercase letters `a` to +`z` and the digits `2` to `7`. + +`rtime`:: + +The time at which the guest was registered in this cluster, in milliseconds +since the UNIX epoch. Creating or cloning a guest, restoring it to a free VMID, +and migrating it from another cluster register the guest in the cluster. A +guest restored to a free VMID gets a new `rtime`, even if it keeps the backup's +`uid`. See xref:guest_identity_lifecycle[Lifecycle] for the other operations. +As a timestamp, the `rtime` is not guaranteed to be unique. + +{pve} manages the `meta` property automatically. It is read-only through the +API, even for `root@pam`. + +IMPORTANT: The `uid` helps to correlate a guest with its backups and copies, +but does not prove where a guest comes from. Whoever restores a backup or +migrates a guest into the cluster supplies the `uid` of that guest, which {pve} +takes over if no other guest in the cluster has it. {pve} keeps a `uid` unique +within a cluster only for operations through its API. A configuration file +copied by hand has the same `uid` as the original, and guests in different +clusters can have the same `uid`. + +[[guest_identity_read]] +Reading the Identity +~~~~~~~~~~~~~~~~~~~~ + +The following places show the guest identity. All of them require the +`VM.Audit` privilege on the guest. + +* In the web interface, the 'Summary' panel of the guest shows the `uid` as + 'Unique ID'. The 'Details' button next to it, which has an info icon, opens + the 'Guest Identity' window. This window additionally shows the `rtime` as + 'Registered' and the creation information. +* The guest configuration, for example with `qm config ` or + `pct config `, and through the API paths + `/nodes//qemu//config` and `/nodes//lxc//config`. +* The guest status at the API paths `/nodes//qemu//status/current` + and `/nodes//lxc//status/current`, which contains the `uid`. +* The cluster resources at the API path `/cluster/resources`, which contain the + `uid` and `rtime` of each guest, for example: ++ +---- +# pvesh get /cluster/resources --type vm --output-format json-pretty +---- + +The configuration in a backup contains the `meta` property as well. You can +show it with `pvesm extractconfig `, for example. + +[[guest_identity_lifecycle]] +Lifecycle +~~~~~~~~~ + +The following table shows how operations affect the identity of a guest. + +[width="100%",cols="3,4,2",options="header"] +|=========================================================================== +|Operation |`uid` |`rtime` +|Create, including import |new |current time +|Clone, including linked clones and clones from snapshots |new |current time +|Restore to a free VMID |the backup's `uid` if present and not used by another +guest in the cluster; otherwise new |current time +|Restore to a free VMID with `unique` set |new |current time +|Restore over an existing guest that has a `uid`, with or without `unique` set +|kept, see xref:guest_identity_restore[Restore Over an Existing Guest] |kept +|Restore over an existing guest without a `uid` |as for a free VMID |current +time +|Migration to another cluster, on the target |the source guest's `uid` if no +other guest in the target cluster has it; otherwise new |current time +|Migration to another cluster, on the source if the guest is kept |kept |kept +|Snapshot rollback |kept, not taken from the snapshot |kept +|Migration within the cluster, HA recovery, configuration change, backup, +replication, conversion to a template |kept |kept +|Destroy |removed |removed +|=========================================================================== + +Restores and migrations from another cluster report the outcome in their task +log, in a line starting with `guest identity:`, for example: + +---- +guest identity: backup uid w5qczgiem26qnwzwsoil is in use by guest 100, keeping uid er3n3wqoqgcl362ep3da +---- + +When a restore or a migration from another cluster takes over a `uid`, the guest +first has a newly assigned `uid`. A restore replaces it with the `uid` of the +backup at its end. A migration from another cluster replaces it with the `uid` +of the source guest when the target takes over the configuration, which is +before the migration finishes. Tools that read `/cluster/resources` during such +a task can therefore see a `uid` that disappears afterwards. A changed `uid` +does not mean that the task has finished. + +Guests that existed before the upgrade to {pve} 9.3 get their identity the +first time a node running {pve} 9.3 or later writes their configuration. This +happens, for example, when their configuration changes, when they are backed +up, or when they are migrated to such a node. Their `rtime` is the time of that +write. Backups made before that have no `uid`. + +When a tool restores a guest by removing it and creating a new one through the +API, the new guest gets a new `uid`. To keep the identity, a tool has to change +the existing guest in place, or let {pve} restore the guest. + +[[guest_identity_restore]] +Restore Over an Existing Guest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the existing guest already has a guest identity, restoring a backup over it +keeps the `uid` and `rtime` of that guest. When both the guest and the backup +have a `uid` and the values differ, {pve} treats the backup as belonging to +another guest. This can happen when a VMID was reused. {pve} refuses such a +restore before it overwrites the guest: + +---- +backup belongs to another guest: its uid 'w5qczgiem26qnwzwsoil' differs from uid 'er3n3wqoqgcl362ep3da' of the existing VM 100 - use 'allow-identity-mismatch' to overwrite the VM anyway +---- + +To restore such a backup anyway, set the `allow-identity-mismatch` option. The +existing guest then keeps its identity. In the web interface, select 'Allow +restoring a backup of another guest' in the restore window, which you open from +the 'Backup' panel of the guest. On the command line, use for example: + +---- +# qmrestore --force 1 --allow-identity-mismatch 1 +# pct restore --force 1 --allow-identity-mismatch 1 +---- + +Backups without a `uid`, such as backups made with {pve} versions older than +9.3, are not refused. Container restores from standard input +(`pct restore -`) are an exception: the `uid` of such a backup is only +known once the container has been overwritten. A restore from standard input +over a container with a `uid` therefore always requires +`allow-identity-mismatch`. + +A restore that fails or is stopped can leave the guest behind. If the restore +was killed, the guest also stays locked. Remove the lock with `qm unlock ` +or `pct unlock ` before you retry the restore or remove the guest. + +If the failed restore had already read the configuration of the backup, +`qm pending ` or `pct pending ` lists an entry for `meta` with the +`uid` of the backup. As long as this entry exists, retrying the same restore +over the guest left behind is accepted. Starting the guest or changing its +options removes the entry. The retry is then refused like a restore of another +guest's backup. + +A successful retry takes over the `uid` of the backup and sets a new `rtime`, as +a restore to a free VMID does. The guest keeps its `uid` and `rtime` instead if +another guest in the cluster has the `uid` of the backup, or if the retry sets +`unique`. Restores from standard input still require `allow-identity-mismatch`, +as described above. + +If the failed restore stopped before it read the backup, for example because the +backup storage was unreachable, the guest left behind is one of the following: + +* An empty placeholder of a restore to a free VMID. Unlock it as described + above, remove it, for example with `qm destroy ` or + `pct destroy `, and restore again. +* An existing guest that had no `uid` before the restore, now with a new `uid`. + Retry with `allow-identity-mismatch`. + +A restore can also complete with a warning starting with +`guest identity: unable to adopt backup uid`, for example if the cluster has no +quorum at the end of the restore. The guest is then restored completely, but +keeps its new `uid`. To take over the `uid` of the backup, repeat the restore +over the guest before you start it or change its configuration. + +[[guest_identity_mixed_versions]] +Mixed Versions in a Cluster +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +While some nodes of a cluster still run a {pve} version older than 9.3, +migrating or cloning a guest to one of these nodes is refused. The guest keeps +running on its current node. If the target node reports its version, the error +message shows that version and the minimum required version. If the node does +not report its version, the operation is refused because {pve} cannot verify +that the node supports guest identities. The 'Migrate' window of the web +interface shows the reason for the selected target node. + +Moves of HA-managed guests to such nodes are refused in the same way. A running +VM that the HA stack migrates keeps running. The HA stack stops a container, and +a VM that it relocates, before it migrates the guest, so such a guest is stopped +although the move is then refused. The HA stack retries automatic moves until +they succeed, and such a guest can stay stopped in the meantime. This includes +moves for: + +* the maintenance mode, +* a node shutdown with the `migrate` shutdown policy, +* node affinity rules, +* the CRS load balancer. + +Such moves complete only after the target nodes that the HA stack selects run +{pve} 9.3 or later. + +To avoid refused moves while you upgrade a cluster, do one of the following: + +* Move all guests away from a node before you upgrade it. +* Upgrade the packages on all nodes, one node at a time, before you reboot any + node or enable the maintenance mode. + +The reason for these refusals is that {pve} versions older than 9.3 ignore the +whole `meta` property of a guest that has a guest identity, including the +creation information. A VM started on such a node can therefore get different +virtual hardware, see xref:qm_machine_type[Machine Type]. When such a node +writes the configuration, the property is lost. + +The check covers migrations and clones started on a node running {pve} 9.3 or +later. It does not cover: + +* the HA recovery of guests from a fenced node, +* configuration files moved by hand, +* operations started on a node with an older version. + +If an older node writes the configuration of a guest in one of these cases, the +guest loses its `meta` property. It gets a new identity the next time a node +running {pve} 9.3 or later writes its configuration. + +Backups made with {pve} 9.3 or later contain the guest identity. Restore them +only on nodes running {pve} 9.3 or later, because older versions ignore the +`meta` property of such backups in the same way. + +A migration to another cluster (`qm remote-migrate`, `pct remote-migrate`) is +refused right after connecting to a target node with a version older than 9.3, +before any data is transferred. The error is +`Remote tunnel endpoint too old, upgrade required`. + [[vzdump_configuration]] Configuration ------------- -- 2.47.3