Re: [RFC qemu/qemu-server/proxmox-backup-qemu 0/9] close #6169: backup: pbs: per-target dirty bitmaps

[Fiona Ebner <f.ebner@proxmox.com>]

Am 02.09.26 um 4:34 PM schrieb Dominik Csapak:
> On 9/2/26 3:56 PM, Fiona Ebner wrote:
>> To decide whether an incremental backup can be done, the checksum of
>> the latest backup on the target is compared against the checksum from
>> the previous backup the QEMU client library did.
>>
>> This was previously tracked per device as the PVE backup code in QEMU
>> also only used a single bitmap per device. So backing up to target A,
>> dirtying the disk and backing up to target B would mean that the
>> checksum for target A is overwritten and a subsequent backup to target
>> A needs to discard the dirty bitmap.
>>
>> Allow for tracking multiple targets with different bitmaps, by saving
>> the checksums per device+target combination, rather than just per
>> device.
>>
>> Whether the new behavior or the old behavior is used is controlled by
>> QEMU, optionally passing along the target ID, and a flag for the
>> export and import functions for migration. These are machine version
>> guarded on the QEMU/qemu-server side to avoid a mismatch.
>>
>>
>> This is an ABI change for the backup library, so a versioned breaks
>> and a versioned depends in the other direction is needed.
>>
> 
> i guess it's not really necessary, but couldn't we add new methods
> instead of modifying the existing ones? then the breaks pve-qemu-kvm
> wouldn't be necessary? (i think?)
> 
> not that important since we must bump both anyway

Yeah, we could add wrappers and then later drop the old methods with a
major PVE version.

>>
>> QUESTION: with old qemu-server, target ID is not provided, but the new
>> machine version might be in use. In this case, the dirty bitmap state
>> is not migrated, in case of a new -> old -> new migration, which can
>> lead to stale info! Would it be okay to add a Breaks for old
>> qemu-server or should I try to resolve it differently somehow?
> 
> is such migration to an older qemu-server even supported?

It is supported in the sense that it can (and will ;) be done in
practice. With the Breaks, we'd avoid that a new machine version
migrates to a node with older qemu-server.

> IIUC the issue is that the
> 1. there is a bitmap with a target
> 2. old qemu-server makes a backup without a target creating
>    a new dirty-bitmap (actually invalidating the old one, but
>    since the code doesn't know about it doesn't do anything with
>    it)
> 3. the new code uses the target name again, bitmap still existing
> 
> the checksum would be different though and it would not be an
> incremental backup once, or?

Yes, it will lead to full backups being necessary each time we migrate
to a node with an old qemu-server (or from a node with old qemu-server
if a backup was made, and if there were changes), because with the new
machine version, the PREVIOUS_CSUMS_PER_TARGET is migrated, but not the
PREVIOUS_CSUMS hash. We could migrate the old hash as well, which would
keep old qemu-server <-> old qemu-server working, but old <-> new would
still mean full backups, so not sure it's worth it.

It is breaking an existing working feature (dirty bitmaps not
successfully re-used with the same PBS target for all online migrations
anymore) and is also intransparent from a user perspective. The Breaks
feels like a clean solution and avoids all problematic cases.

> 
> question is, should we support both modes simultaniously at all?
> we could require a target always, or if no target is given,
> invalidate all named target bitmaps.

That's my suggestion with the Breaks for old qemu-server. If we do that,
then new machine version means target ID is there and old machine
version means no target ID in all cases. The latter part is important
for the existing feature to continue working in all cases even with
migration to and from old versions.

> 
> also, the same can happen with new versions only too?
> backup to target storage X
> on pve add identical storage Y -> backup to same storage as
> different id
> backup with X again -> stale bitmap?

It's a different storage from PVE's perspective so that is fine IMHO. It
could be solved: rather than the PVE storage ID, the server identity +
namespace combination could be used as the target ID. But then the
approach with dropping stale bitmaps according to the storage config
becomes more involved, as we'd need to query all PBS entries for the
identity. I'm a bit unsure, because such a config is actually
unsupported (e.g. no locking guarantee), but I will think about it.

> 
>>
>> QUESTION: How to best avoid 'orphaned' bitmaps? Auto-remove for
>> storages no longer in configuration? Or have some knob/limit for how
>> many to keep?
>>
> 
> i think it depends how big these are? I doubt many users
> make many backups to storages they abandon later

The bitmap granularity is 4 MiB, so for a 4 TiB disk that's 4 TiB / 4
MiB = 1 MiB worth of bits to keep track of. So the overhead should not
be too bad in practice, but I'll test around a bit and see.

I was considering the "don't do anything" approach, but since the bitmap
state is migrated and especially after feature #3233 for persisting the
dirty bitmaps, it will not be dropped "naturally" often.

But if the overhead is negligible, maybe I'll still just go for the
"don't do anything" approach to start out. Dropping based on storage
config can be added later.

Dropping based on some other limit might require more info to be saved,
maybe the timestamp in the PREVIOUS_CSUMS hash values, so we could drop
the oldest? To avoid another ABI change in the future, maybe I should
already add it? Not sure, I kinda prefer the storage config approach.

> also how could we differ between a user just recreating
> a storage entry (for whatever reason) and a genuine
> deletion? (if the backup timing is right)

Well, we can't distinguish, but I think this is rare enough in practice
and not too unexpected from a user perspective that after recreating, a
full backup is done.