From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 972DC1FF09F for ; Thu, 03 Sep 2026 13:09:33 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 615D721576; Thu, 03 Sep 2026 13:09:33 +0200 (CEST) Message-ID: <0fbe752e-97a8-4180-a863-47916051a6b8@proxmox.com> Date: Thu, 3 Sep 2026 13:09:25 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC qemu/qemu-server/proxmox-backup-qemu 0/9] close #6169: backup: pbs: per-target dirty bitmaps To: Dominik Csapak , pve-devel@lists.proxmox.com References: <20260902135536.525194-1-f.ebner@proxmox.com> <61552bba-c898-4ea7-bd32-82bc5daac8a9@proxmox.com> Content-Language: en-US From: Fiona Ebner In-Reply-To: <61552bba-c898-4ea7-bd32-82bc5daac8a9@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1788433767198 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.731 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: MXWJAKFACDDXS75R4IABEFYL3IX46FDR X-Message-ID-Hash: MXWJAKFACDDXS75R4IABEFYL3IX46FDR X-MailFrom: f.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Am 02.09.26 um 4:34 PM schrieb Dominik Csapak: > On 9/2/26 3:56 PM, Fiona Ebner wrote: >> To decide whether an incremental backup can be done, the checksum of >> the latest backup on the target is compared against the checksum from >> the previous backup the QEMU client library did. >> >> This was previously tracked per device as the PVE backup code in QEMU >> also only used a single bitmap per device. So backing up to target A, >> dirtying the disk and backing up to target B would mean that the >> checksum for target A is overwritten and a subsequent backup to target >> A needs to discard the dirty bitmap. >> >> Allow for tracking multiple targets with different bitmaps, by saving >> the checksums per device+target combination, rather than just per >> device. >> >> Whether the new behavior or the old behavior is used is controlled by >> QEMU, optionally passing along the target ID, and a flag for the >> export and import functions for migration. These are machine version >> guarded on the QEMU/qemu-server side to avoid a mismatch. >> >> >> This is an ABI change for the backup library, so a versioned breaks >> and a versioned depends in the other direction is needed. >> > > i guess it's not really necessary, but couldn't we add new methods > instead of modifying the existing ones? then the breaks pve-qemu-kvm > wouldn't be necessary? (i think?) > > not that important since we must bump both anyway Yeah, we could add wrappers and then later drop the old methods with a major PVE version. >> >> QUESTION: with old qemu-server, target ID is not provided, but the new >> machine version might be in use. In this case, the dirty bitmap state >> is not migrated, in case of a new -> old -> new migration, which can >> lead to stale info! Would it be okay to add a Breaks for old >> qemu-server or should I try to resolve it differently somehow? > > is such migration to an older qemu-server even supported? It is supported in the sense that it can (and will ;) be done in practice. With the Breaks, we'd avoid that a new machine version migrates to a node with older qemu-server. > IIUC the issue is that the > 1. there is a bitmap with a target > 2. old qemu-server makes a backup without a target creating >    a new dirty-bitmap (actually invalidating the old one, but >    since the code doesn't know about it doesn't do anything with >    it) > 3. the new code uses the target name again, bitmap still existing > > the checksum would be different though and it would not be an > incremental backup once, or? Yes, it will lead to full backups being necessary each time we migrate to a node with an old qemu-server (or from a node with old qemu-server if a backup was made, and if there were changes), because with the new machine version, the PREVIOUS_CSUMS_PER_TARGET is migrated, but not the PREVIOUS_CSUMS hash. We could migrate the old hash as well, which would keep old qemu-server <-> old qemu-server working, but old <-> new would still mean full backups, so not sure it's worth it. It is breaking an existing working feature (dirty bitmaps not successfully re-used with the same PBS target for all online migrations anymore) and is also intransparent from a user perspective. The Breaks feels like a clean solution and avoids all problematic cases. > > question is, should we support both modes simultaniously at all? > we could require a target always, or if no target is given, > invalidate all named target bitmaps. That's my suggestion with the Breaks for old qemu-server. If we do that, then new machine version means target ID is there and old machine version means no target ID in all cases. The latter part is important for the existing feature to continue working in all cases even with migration to and from old versions. > > also, the same can happen with new versions only too? > backup to target storage X > on pve add identical storage Y -> backup to same storage as > different id > backup with X again -> stale bitmap? It's a different storage from PVE's perspective so that is fine IMHO. It could be solved: rather than the PVE storage ID, the server identity + namespace combination could be used as the target ID. But then the approach with dropping stale bitmaps according to the storage config becomes more involved, as we'd need to query all PBS entries for the identity. I'm a bit unsure, because such a config is actually unsupported (e.g. no locking guarantee), but I will think about it. > >> >> QUESTION: How to best avoid 'orphaned' bitmaps? Auto-remove for >> storages no longer in configuration? Or have some knob/limit for how >> many to keep? >> > > i think it depends how big these are? I doubt many users > make many backups to storages they abandon later The bitmap granularity is 4 MiB, so for a 4 TiB disk that's 4 TiB / 4 MiB = 1 MiB worth of bits to keep track of. So the overhead should not be too bad in practice, but I'll test around a bit and see. I was considering the "don't do anything" approach, but since the bitmap state is migrated and especially after feature #3233 for persisting the dirty bitmaps, it will not be dropped "naturally" often. But if the overhead is negligible, maybe I'll still just go for the "don't do anything" approach to start out. Dropping based on storage config can be added later. Dropping based on some other limit might require more info to be saved, maybe the timestamp in the PREVIOUS_CSUMS hash values, so we could drop the oldest? To avoid another ABI change in the future, maybe I should already add it? Not sure, I kinda prefer the storage config approach. > also how could we differ between a user just recreating > a storage entry (for whatever reason) and a genuine > deletion? (if the backup timing is right) Well, we can't distinguish, but I think this is rare enough in practice and not too unexpected from a user perspective that after recreating, a full backup is done.