public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH docs v3 35/38] vzdump: describe the guest identity of VMs and containers
Date: Wed, 18 Nov 2026 05:37:47 +0100	[thread overview]
Message-ID: <20261118043817.1659212-36-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118043817.1659212-1-t.lamprecht@proxmox.com>

Since Proxmox VE 9.3, VMs and containers have a guest identity in
'meta': a random uid and a registration time. Backup software and other
tools can distinguish a guest from another that reuses its VMID [0].

Administrators need to understand why restores over a different guest
are refused, when a restore keeps the uid of the backup, and why
migrations and clones to older nodes are blocked. Those nodes drop
'meta', including VM creation information used for hardware defaults.

Document the lifecycle, failed-restore retries, and mixed-version
restrictions in the backup and restore chapter, where users are most
likely to encounter the identity. Include migration and HA behavior
there so the rules can be read together.

[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=7846

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v2 -> v3:
* backups are still grouped by type and VMID, also for retention
* refused remote migration: do not remove the reported guest if it is
  the source itself
* own section for failed restores, with what a killed or late-failing
  restore leaves behind
* remote migration of a guest without uid, HA recovery through the node
  of the active CRM, and smaller corrections (the uid gets compared, not
  the identity)

changes v1 -> v2:
* say that the uid has 100 random bits and how rtime differs from ctime
* describe the refusal of a migration to another cluster, its typical
  cases and 'allow-new-uid'
* retry after a failed restore: scope the pending entry, name the
  warning, correct the placeholder case and add the note for enforced
  unique guest IDs

 vzdump.adoc | 332 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 332 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..1d74602 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -464,6 +464,13 @@ following CLI tools:
 
 For details see the corresponding manual pages.
 
+When you restore a backup over an existing guest, {pve} compares the `uid` of
+the backup and of the guest. If they differ, the restore is refused, unless you
+set the `allow-identity-mismatch` option. The existing guest then keeps its
+identity. See
+xref:guest_identity_restore[Restore Over an Existing Guest] for details and
+exceptions.
+
 Bandwidth Limit
 ~~~~~~~~~~~~~~~
 
@@ -563,6 +570,331 @@ file in `/var/log/proxmox-backup/file-restore/`. The log file might contain
 additional information in case an attempt to restore individual files or
 accessing file systems contained in a backup archive fails.
 
+[[guest_identity]]
+Guest Identity
+--------------
+
+Since {pve} 9.3, each virtual machine and container gets a guest identity: a
+random identifier, `uid`, and a registration time, `rtime`. Unlike a VMID,
+which can be reused after a guest is destroyed, the `uid` lets backup software
+and other tools recognize the same logical guest across restores and
+migrations, even under a different VMID. Whether an operation keeps the `uid`
+depends on the target and the options of the operation, as described in
+xref:guest_identity_lifecycle[Lifecycle].
+
+Backups are still grouped by guest type and VMID, also for retention. If a new
+guest reuses the VMID of a removed guest of the same type, pruning handles the
+backups of both guests together.
+
+[[guest_identity_values]]
+Identity Values
+~~~~~~~~~~~~~~~
+
+The guest identity is part of the `meta` property in the guest configuration.
+This property also contains the creation time `ctime` and, for VMs, the QEMU
+version that the VM was created with, `creation-qemu`:
+
+----
+meta: creation-qemu=10.1.2,ctime=1772444467,rtime=1794320571482,uid=w5qczgiem26qnwzwsoil
+----
+
+`uid`::
+
+A random identifier of 20 characters, consisting of the lowercase letters `a` to
+`z` and the digits `2` to `7`. It encodes 100 random bits, so two guests
+practically never get the same `uid` by chance.
+
+`rtime`::
+
+The time at which the guest was registered in this cluster, in milliseconds
+since the UNIX epoch. Creating or cloning a guest, restoring it to a free VMID,
+and migrating it from another cluster register the guest in the cluster. A
+guest restored to a free VMID gets a new `rtime`, even if it keeps the backup's
+`uid`. See xref:guest_identity_lifecycle[Lifecycle] for the other operations.
+As a timestamp, the `rtime` is not guaranteed to be unique.
+
+The `rtime` differs from the creation time `ctime`. The `ctime` stays with the
+guest: a clone has the `ctime` of its source, and a restored guest or a guest
+migrated from another cluster has that of the backup or of the source guest. The
+`rtime` is set whenever the guest gets its `uid` in this cluster. A guest whose
+`uid` is unchanged but whose `rtime` is newer was therefore registered again,
+for example by a restore to a free VMID.
+
+{pve} manages the `meta` property automatically. It is read-only through the
+API, even for `root@pam`.
+
+IMPORTANT: The `uid` helps to correlate a guest with its backups and copies,
+but does not prove where a guest comes from. Whoever restores a backup or
+migrates a guest into the cluster supplies the `uid` of that guest, which {pve}
+takes over if no other guest in the cluster has it. {pve} keeps a `uid` unique
+within a cluster only for operations through its API. A configuration file
+copied by hand has the same `uid` as the original, and guests in different
+clusters can have the same `uid`.
+
+[[guest_identity_read]]
+Reading the Identity
+~~~~~~~~~~~~~~~~~~~~
+
+The following places show the guest identity. All of them require the
+`VM.Audit` privilege on the guest.
+
+* In the web interface, the 'Summary' panel of the guest shows the `uid` as
+  'Unique ID'. The 'Details' button next to it, which has an info icon, opens
+  the 'Guest Identity' window. This window additionally shows the `rtime` as
+  'Registered' and the creation information.
+* The guest configuration, for example with `qm config <vmid>` or
+  `pct config <vmid>`, and through the API paths
+  `/nodes/<node>/qemu/<vmid>/config` and `/nodes/<node>/lxc/<vmid>/config`.
+* The guest status at the API paths `/nodes/<node>/qemu/<vmid>/status/current`
+  and `/nodes/<node>/lxc/<vmid>/status/current`, which contains the `uid`.
+* The cluster resources at the API path `/cluster/resources`, which contain the
+  `uid` and `rtime` of each guest, for example:
++
+----
+# pvesh get /cluster/resources --type vm --output-format json-pretty
+----
+
+The configuration in a backup contains the `meta` property as well. You can
+show it with `pvesm extractconfig <volume>`, for example.
+
+[[guest_identity_lifecycle]]
+Lifecycle
+~~~~~~~~~
+
+The following table shows how operations affect the identity of a guest.
+
+[width="100%",cols="3,4,2",options="header"]
+|===========================================================================
+|Operation |`uid` |`rtime`
+|Create, including import |new |current time
+|Clone, including linked clones and clones from snapshots |new |current time
+|Restore to a free VMID |the backup's `uid` if present and not used by another
+guest in the cluster; otherwise new |current time
+|Restore to a free VMID with `unique` set |new |current time
+|Restore over an existing guest that has a `uid`, with or without `unique` set
+|kept, see xref:guest_identity_restore[Restore Over an Existing Guest] |kept
+|Restore over an existing guest without a `uid` |as for a free VMID |current
+time
+|Migration to another cluster, on the target |the source guest's `uid`; new if
+another guest in the target cluster has it and `allow-new-uid` is set, or if the
+source node runs a {pve} version older than 9.3, see
+xref:guest_identity_remote_migration[Migration to Another Cluster] |current time
+|Migration to another cluster, on the source if the guest is kept |kept |kept
+|Snapshot rollback |kept, not taken from the snapshot |kept
+|Migration within the cluster, HA recovery, configuration change, backup,
+replication, conversion to a template |kept |kept
+|Destroy |removed |removed
+|===========================================================================
+
+Restores and migrations from another cluster report the outcome in their task
+log, in a line with `guest identity:`, for example:
+
+----
+guest identity: backup uid w5qczgiem26qnwzwsoil is in use by guest 100, keeping uid er3n3wqoqgcl362ep3da
+----
+
+When a restore takes over a `uid`, the guest first has a newly assigned `uid`,
+which the restore replaces with the `uid` of the backup at its end. Tools that
+read `/cluster/resources` during a restore can therefore see a `uid` that
+disappears afterwards, also on a guest that is no longer locked. A migration
+from another cluster takes over the `uid` of the source guest right at its
+start. In both cases, a changed `uid` does not mean that the task has finished.
+
+Guests that existed before the upgrade to {pve} 9.3 get their identity the
+first time a node running {pve} 9.3 or later writes their configuration. This
+happens, for example, when their configuration changes, when they are backed
+up, or when they are migrated to such a node. Their `rtime` is the time of that
+write. Backups made before that have no `uid`.
+
+When a tool restores a guest by removing it and creating a new one through the
+API, the new guest gets a new `uid`. To keep the identity, a tool has to change
+the existing guest in place, or let {pve} restore the guest.
+
+[[guest_identity_restore]]
+Restore Over an Existing Guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the existing guest already has a guest identity, restoring a backup over it
+keeps the `uid` and `rtime` of that guest. When both the guest and the backup
+have a `uid` and the values differ, {pve} treats the backup as belonging to
+another guest. This can happen when a VMID was reused. {pve} refuses such a
+restore before it overwrites the guest:
+
+----
+backup belongs to another guest: its uid 'w5qczgiem26qnwzwsoil' differs from uid 'er3n3wqoqgcl362ep3da' of the existing VM 100 - use 'allow-identity-mismatch' to overwrite the VM anyway
+----
+
+To restore such a backup anyway, set the `allow-identity-mismatch` option. The
+existing guest then keeps its identity. In the web interface, select 'Allow
+restoring a backup of another guest' in the restore window, which you open from
+the 'Backup' panel of the guest. On the command line, use for example:
+
+----
+# qmrestore <archive> <vmid> --force 1 --allow-identity-mismatch 1
+# pct restore <vmid> <archive> --force 1 --allow-identity-mismatch 1
+----
+
+Backups without a `uid`, such as backups made with {pve} versions older than
+9.3, are not refused. Container restores from standard input
+(`pct restore <vmid> -`) are an exception: the `uid` of such a backup is only
+known once the container has been overwritten. A restore from standard input
+over a container with a `uid` therefore always requires
+`allow-identity-mismatch`.
+
+[[guest_identity_failed_restore]]
+Failed Restores
+~~~~~~~~~~~~~~~
+
+A restore that fails or is stopped can leave the guest behind, for example a
+restore over an existing guest, or a VM restore that fails after it restored the
+data or in the live phase of a live restore. A container restore over an
+existing container that fails after it removed the old volumes removes the
+container. If the restore task was killed, the guest can also stay locked.
+Remove the lock with `qm unlock <vmid>` or `pct unlock <vmid>` before you retry
+the restore or remove the guest.
+
+If the failed restore was to a free VMID or over a guest without a `uid`, and it
+had already read a backup that has a `uid`, `qm pending <vmid>` or
+`pct pending <vmid>` lists an entry for `meta` with the `uid` of the backup. As
+long as this entry exists, retrying the same restore over the guest left behind
+is accepted. Starting the guest or changing its options removes the entry and
+logs a warning that names the `uid`. The retry is then refused like a restore of
+another guest's backup.
+
+A successful retry takes over the `uid` of the backup and sets a new `rtime`, as
+a restore to a free VMID does. The guest keeps its `uid` and `rtime` instead if
+another guest in the cluster has the `uid` of the backup, or if the retry sets
+`unique`. Restores from standard input still require `allow-identity-mismatch`,
+as described above.
+
+A guest that had a `uid` and no such entry before the failed restore keeps its
+`uid` and gets no entry. A retry is checked like the first attempt: a backup
+with the same `uid` is accepted, also after the guest was started or changed,
+and a backup with another `uid` requires `allow-identity-mismatch`.
+
+A restore to a free VMID that fails before it restored the data normally
+removes the guest again. If the restore task was killed, or if that removal
+failed, an empty placeholder can stay at that VMID instead. Unlock it as
+described above if it is locked, and remove it, for example with
+`qm destroy <vmid>` or `pct destroy <vmid>`. Add
+`--destroy-unreferenced-disks 1` to also remove volumes that a killed restore
+created for that VMID. In both cases, restore again. Use
+another VMID if the xref:pvecm_next_id_range[`next-id`] setting of the
+datacenter has the `unique` and `enforce` options set, as the first VMID then
+counts as used.
+
+If a restore over an existing guest without a `uid` stopped before it read the
+backup, for example because the backup storage was unreachable, the guest now
+has a new `uid`. Retry with `allow-identity-mismatch`. The guest then keeps the
+new `uid`.
+
+A restore can also complete with a warning starting with
+`guest identity: unable to adopt backup uid`, for example if the cluster-wide
+lock for this step could not be taken in time. The guest is then restored
+completely, but keeps its new `uid`. To take over the `uid` of the backup,
+repeat the restore over the guest before you start it or change its
+configuration.
+
+[[guest_identity_remote_migration]]
+Migration to Another Cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A migration to another cluster (`qm remote-migrate`, `pct remote-migrate`) keeps
+the `uid` of the guest. The target takes it over at the start of the migration,
+before any data is transferred.
+
+If another guest in the target cluster already has that `uid`, the migration is
+refused at that point. Typical cases are:
+
+* the guest gets migrated back to a cluster that still has the source guest of
+  an earlier migration, which stays there unless `delete` was set,
+* a copy of the guest was restored in the target cluster from a backup,
+* an earlier migration of the guest broke off after it had transferred the
+  configuration and left a locked guest in the target cluster,
+* the target is the cluster that the guest is in.
+
+The error ends with:
+
+----
+guest 105 in the target cluster already has the uid 'w5qczgiem26qnwzwsoil' of this guest - use 'allow-new-uid' to migrate it with a new uid
+----
+
+If the reported guest is the source guest itself, do not remove it. To keep the
+`uid`, migrate the guest within the cluster instead, with `qm migrate` or
+`pct migrate`.
+
+Otherwise, to keep the `uid`, remove the other guest in the target cluster and
+migrate again. To migrate anyway, set the `allow-new-uid` option. The migrated
+guest then gets a new `uid`, and the other guest keeps the old one. In both
+cases, use another target VMID if the xref:pvecm_next_id_range[`next-id`]
+setting of the target cluster has the `unique` and `enforce` options set, as the
+refused migration already used the first one. For example:
+
+----
+# qm remote-migrate <vmid> <target-vmid> <target-endpoint> --target-bridge <bridge> --target-storage <storage> --allow-new-uid 1
+----
+
+[[guest_identity_mixed_versions]]
+Mixed Versions in a Cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+While some nodes of a cluster still run a {pve} version older than 9.3,
+migrating or cloning a guest to one of these nodes is refused. The guest keeps
+running on its current node. If the target node reports its version, the error
+message shows that version and the minimum required version. If the node does
+not report its version, the operation is refused because {pve} cannot verify
+that the node supports guest identities. The 'Migrate' window of the web
+interface shows the reason for the selected target node.
+
+Moves of HA-managed guests to such nodes are refused in the same way. A running
+VM that the HA stack migrates keeps running. The HA stack stops a container, and
+a VM that it relocates, before it migrates the guest, so such a guest is stopped
+although the move is then refused. The HA stack retries automatic moves as long
+as it selects such a node, and such a guest can stay stopped in the meantime.
+This includes moves for:
+
+* the maintenance mode,
+* a node shutdown with the `migrate` shutdown policy,
+* node affinity rules,
+* the CRS load balancer.
+
+Such moves complete only after the target nodes that the HA stack selects run
+{pve} 9.3 or later.
+
+To avoid refused moves while you upgrade a cluster, do one of the following:
+
+* Move all guests away from a node before you upgrade it.
+* Upgrade the packages on all nodes, one node at a time, before you reboot any
+  node or enable the maintenance mode.
+
+The reason for these refusals is that {pve} versions older than 9.3 ignore the
+whole `meta` property of a guest that has a guest identity, including the
+creation information. A VM started on such a node can therefore get different
+virtual hardware, see xref:qm_machine_type[Machine Type]. When such a node
+writes the configuration, the property is lost.
+
+The check covers migrations and clones started on a node running {pve} 9.3 or
+later. It does not cover:
+
+* the HA recovery of guests from a fenced node,
+* configuration files moved by hand,
+* operations started on a node with an older version.
+
+If an older node writes the configuration of a guest in one of these cases, the
+guest loses its `meta` property. In an HA recovery, such a node can also be the
+node of the active HA manager (CRM), which writes the configuration when it
+removes a leftover lock, for example of a backup. The guest gets a new identity
+the next time a node running {pve} 9.3 or later writes its configuration.
+
+Backups made with {pve} 9.3 or later contain the guest identity. Restore them
+only on nodes running {pve} 9.3 or later, because older versions ignore the
+`meta` property of such backups in the same way.
+
+A migration to another cluster (`qm remote-migrate`, `pct remote-migrate`) is
+refused right after connecting to a target node with a version older than 9.3,
+before any data is transferred. The error is
+`Remote tunnel endpoint too old, upgrade required`.
+
 [[vzdump_configuration]]
 Configuration
 -------------
-- 
2.47.3





  parent reply	other threads:[~2026-11-18  4:43 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  4:37 [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 01/38] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 02/38] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 03/38] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 04/38] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 05/38] pmxcfs: confdb: call an optional listener when the nodes get published Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 06/38] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 07/38] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 08/38] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 09/38] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 10/38] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 11/38] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 12/38] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 13/38] migrate: add shared helpers for the identity command of the migration tunnel Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 14/38] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 15/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 16/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 17/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 18/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 19/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 20/38] tests: remote migration: check a failed command and an early tunnel end Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 21/38] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 22/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 23/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 24/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 25/38] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 26/38] test: remote migration: check the cleanup of the target after failures Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 27/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 28/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 29/38] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 30/38] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 31/38] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 32/38] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 33/38] ui: guest summary: show " Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 34/38] api: bulk migrate: refuse targets that would drop " Thomas Lamprecht
2026-11-18  4:37 ` Thomas Lamprecht [this message]
2026-11-18  4:37 ` [PATCH docs v3 36/38] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 37/38] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 38/38] pct: " Thomas Lamprecht
2026-11-18 13:49 ` [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Dominik Csapak
2026-11-18 14:13   ` Thomas Lamprecht
2026-11-19 10:38 ` superseded: " Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118043817.1659212-36-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal