* [PATCH storage 1/3] lvm: avoid qcow2 images outgrowing their LV after offline snapshot delete
2026-11-13 1:17 [PATCH storage 0/3] lvm: fix qcow2 images outgrowing their LV Thomas Lamprecht
@ 2026-11-13 1:17 ` Thomas Lamprecht
2026-11-13 1:17 ` [PATCH storage 2/3] lvm: qcow2 resize: size the LV for the cluster settings of the image Thomas Lamprecht
` (2 subsequent siblings)
3 siblings, 0 replies; 8+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 1:17 UTC (permalink / raw)
To: pve-devel
Deleting the oldest snapshot of a stopped VM commits the current image
into the snapshot image with 'qemu-img commit'. For every range that
the overlay holds as zeroes, the commit job writes zeroes with unmap to
the base. Such ranges come from guest discards, but also from zero
writes detected by the default 'detect-zeroes=on'. Without the
'discard-no-unref' option, qcow2 then drops the preallocated clusters
of the base image that these ranges fully cover.
Guest writes to such ranges later need newly allocated clusters. The
allocator searches forward from a cursor that only moves back when
clusters get freed, and skips holes that are too small for a request.
So the image can grow beyond the size reported by 'qemu-img measure',
and thus beyond its LV, while free clusters remain inside it. Writes
then fail with ENOSPC, which pauses the VM with the default error
policy.
Set 'discard-no-unref' for the base image of the commit, as qemu-server
commit a3a9a2ab ("fix #6543: use qcow2 'discard-no-unref' option when
using snapshot-as-volume-chain") already does for running VMs and
'qemu-img convert'. Images that already lost clusters stay affected
until they are moved to a newly allocated volume.
Fixes: eda88c94 ("lvmplugin: add qcow2 snapshot")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Storage/LVMPlugin.pm | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index 1b0c8020..d9845e4d 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -1910,7 +1910,10 @@ sub volume_snapshot_delete {
print "running 'qemu-img commit $childpath'\n";
#can't use -d here, as it's an lvm volume
- $cmd = ['/usr/bin/qemu-img', 'commit', '--', $childpath];
+ # keep preallocated base clusters, freeing them lets the image outgrow the LV
+ my $opts = "driver=qcow2,file.driver=host_device,file.filename=$childpath"
+ . ",backing.discard-no-unref=on";
+ $cmd = ['/usr/bin/qemu-img', 'commit', '--image-opts', '--', $opts];
eval { run_command($cmd) };
if ($@) {
warn
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH storage 2/3] lvm: qcow2 resize: size the LV for the cluster settings of the image
2026-11-13 1:17 [PATCH storage 0/3] lvm: fix qcow2 images outgrowing their LV Thomas Lamprecht
2026-11-13 1:17 ` [PATCH storage 1/3] lvm: avoid qcow2 images outgrowing their LV after offline snapshot delete Thomas Lamprecht
@ 2026-11-13 1:17 ` Thomas Lamprecht
2026-11-13 1:17 ` [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps Thomas Lamprecht
2026-11-13 9:52 ` applied: (subset) [PATCH storage 0/3] lvm: fix qcow2 images outgrowing their LV Fiona Ebner
3 siblings, 0 replies; 8+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 1:17 UTC (permalink / raw)
To: pve-devel
When resizing, the LV size was always measured for 64 KiB clusters
without extended L2 entries, which are the settings of base images.
Snapshot overlays use 128 KiB clusters with extended L2 entries, for
which 'qemu-img measure' reports a different size. For small overlays,
the result could be a few hundred KiB too small, which the rounding up
to full extents usually, but not always, covers. For large overlays,
the LV got bigger than needed, by 14 MiB for 900 GiB for example.
Measure with the cluster settings of the image that is resized.
Fixes: eda88c94 ("lvmplugin: add qcow2 snapshot")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Storage/LVMPlugin.pm | 29 ++++++++++++++++++++++-------
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index d9845e4d..6b9867f3 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -1303,14 +1303,12 @@ my sub lvm_qcow2_format {
}
my sub calculate_lvm_size {
- my ($size, $fmt, $backing_snap) = @_;
+ my ($size, $fmt, $measure_options) = @_;
#input size = qcow2 image size in kb
return $size if $fmt ne 'qcow2';
- my $options = $backing_snap ? ['extended_l2=on', 'cluster_size=128k'] : [];
-
- my $json = PVE::Storage::Common::qemu_img_measure($size, $fmt, 5, $options);
+ my $json = PVE::Storage::Common::qemu_img_measure($size, $fmt, 5, $measure_options);
die "failed to query file information with qemu-img measure\n" if !$json;
my $info = eval { decode_json($json) };
if ($@) {
@@ -1322,6 +1320,21 @@ my sub calculate_lvm_size {
return $info->{'fully-allocated'} / 1024;
}
+# snapshot overlays use other cluster settings than base images, which changes the measured size
+my sub qcow2_measure_options {
+ my ($path) = @_;
+
+ my $json = PVE::Storage::Common::qemu_img_info($path, 'qcow2', 10);
+ my $info = eval { decode_json($json) } // {};
+ my $cluster_size = $info->{'cluster-size'}
+ or die "unable to determine the cluster size of '$path'\n";
+
+ my $options = ["cluster_size=$cluster_size"];
+ push $options->@*, 'extended_l2=on' if $info->{'format-specific'}->{data}->{'extended-l2'};
+
+ return $options;
+}
+
my sub alloc_lvm_image {
my ($class, $storeid, $scfg, $vmid, $fmt, $name, $size, $backing_snap) = @_;
@@ -1340,7 +1353,8 @@ my sub alloc_lvm_image {
die "no such volume group '$vg'\n" if !defined($vgs->{$vg});
my $free = int($vgs->{$vg}->{free});
- my $lvmsize = calculate_lvm_size($size, $fmt, $backing_snap);
+ my $measure_options = $backing_snap ? ['extended_l2=on', 'cluster_size=128k'] : [];
+ my $lvmsize = calculate_lvm_size($size, $fmt, $measure_options);
die "not enough free space ($free < $size)\n" if $free < $size;
@@ -1691,10 +1705,11 @@ sub volume_resize {
my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $format) =
$class->parse_volname($volname);
- my $lvmsize = calculate_lvm_size($size / 1024, $format);
-
my $path = $class->path($scfg, $volname, $storeid, $snapname);
+ my $measure_options = $format eq 'qcow2' ? qcow2_measure_options($path) : undef;
+ my $lvmsize = calculate_lvm_size($size / 1024, $format, $measure_options);
+
# For qcow2, the expected size of the LV might get out of sync, so check.
my ($skip_lvextend, $old_lvmsize);
if ($format eq 'qcow2') {
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps
2026-11-13 1:17 [PATCH storage 0/3] lvm: fix qcow2 images outgrowing their LV Thomas Lamprecht
2026-11-13 1:17 ` [PATCH storage 1/3] lvm: avoid qcow2 images outgrowing their LV after offline snapshot delete Thomas Lamprecht
2026-11-13 1:17 ` [PATCH storage 2/3] lvm: qcow2 resize: size the LV for the cluster settings of the image Thomas Lamprecht
@ 2026-11-13 1:17 ` Thomas Lamprecht
2026-11-13 9:54 ` Fiona Ebner
2026-11-13 12:33 ` applied: " Fiona Ebner
2026-11-13 9:52 ` applied: (subset) [PATCH storage 0/3] lvm: fix qcow2 images outgrowing their LV Fiona Ebner
3 siblings, 2 replies; 8+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 1:17 UTC (permalink / raw)
To: pve-devel
Backup solutions like Veeam add persistent dirty bitmaps to qcow2
images to track changed blocks. QEMU stores them inside the image when
the VM is stopped or migrated, in addition to what 'qemu-img measure'
reports and what the LV size is based on. So only the space from
rounding up to full extents was left for them, which depends on the
image size and can be close to nothing. As QEMU writes all new bitmap
clusters before freeing the old ones, a store needs room for two
copies, almost 4 MiB for one bitmap on a 900 GiB image. If that space
is missing, storing fails with ENOSPC and the bitmap is lost.
Reserve space for two copies of two fully dirty bitmaps at the default
granularity of 64 KiB, which is also what Veeam was observed to use,
plus a refcount block for them. That is about 8 KiB per GiB of disk
size, for example 8.4 MiB for 900 GiB. The calculation assumes 128 KiB
clusters, so it also covers snapshot overlays. A finer granularity or
more than two bitmaps can still exceed the reserved space. So can a
store that needs to grow the refcount table, which happens at least for
images ending just below 16 TiB with 64 KiB clusters or 128 TiB with
128 KiB clusters, as QEMU then places the new refcount structures at the
next refcount block boundary, up to 2 GiB or 8 GiB further.
Existing LVs only get the additional space when they are resized.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Storage/LVMPlugin.pm | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index 6b9867f3..e576bd74 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -9,6 +9,7 @@ use File::Basename;
use IO::File;
use JSON;
use List::Util qw(max);
+use POSIX qw(ceil);
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
use PVE::Diskmanage;
@@ -1302,6 +1303,19 @@ my sub lvm_qcow2_format {
}
}
+# room for two copies (old and new while storing) of two fully dirty persistent bitmaps at 64 KiB
+# granularity, and a refcount block for them, assuming 128 KiB clusters
+my sub persistent_bitmaps_headroom {
+ my ($size) = @_; # in KiB
+
+ my $cluster_kib = 128;
+ my $data_clusters = ceil($size / 64 / 8 / ($cluster_kib * 1024));
+ my $table_clusters = ceil($data_clusters * 8 / ($cluster_kib * 1024));
+ my $copy_clusters = 2 * ($data_clusters + $table_clusters) + 1; # one directory cluster
+
+ return (2 * $copy_clusters + 1) * $cluster_kib;
+}
+
my sub calculate_lvm_size {
my ($size, $fmt, $measure_options) = @_;
#input size = qcow2 image size in kb
@@ -1317,7 +1331,7 @@ my sub calculate_lvm_size {
die "Missing fully-allocated value from json" if !$info->{'fully-allocated'};
- return $info->{'fully-allocated'} / 1024;
+ return $info->{'fully-allocated'} / 1024 + persistent_bitmaps_headroom($size);
}
# snapshot overlays use other cluster settings than base images, which changes the measured size
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps
2026-11-13 1:17 ` [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps Thomas Lamprecht
@ 2026-11-13 9:54 ` Fiona Ebner
2026-11-13 11:12 ` Thomas Lamprecht
2026-11-13 12:33 ` applied: " Fiona Ebner
1 sibling, 1 reply; 8+ messages in thread
From: Fiona Ebner @ 2026-11-13 9:54 UTC (permalink / raw)
To: Thomas Lamprecht, pve-devel
Am 13.11.26 um 2:27 AM schrieb Thomas Lamprecht:
> Backup solutions like Veeam add persistent dirty bitmaps to qcow2
> images to track changed blocks. QEMU stores them inside the image when
As others already pointed out, this is not true anymore.
> the VM is stopped or migrated, in addition to what 'qemu-img measure'
> reports and what the LV size is based on. So only the space from
> rounding up to full extents was left for them, which depends on the
> image size and can be close to nothing. As QEMU writes all new bitmap
> clusters before freeing the old ones, a store needs room for two
> copies, almost 4 MiB for one bitmap on a 900 GiB image. If that space
> is missing, storing fails with ENOSPC and the bitmap is lost.
>
> Reserve space for two copies of two fully dirty bitmaps at the default
> granularity of 64 KiB, which is also what Veeam was observed to use,
> plus a refcount block for them. That is about 8 KiB per GiB of disk
> size, for example 8.4 MiB for 900 GiB. The calculation assumes 128 KiB
> clusters, so it also covers snapshot overlays. A finer granularity or
> more than two bitmaps can still exceed the reserved space. So can a
> store that needs to grow the refcount table, which happens at least for
> images ending just below 16 TiB with 64 KiB clusters or 128 TiB with
> 128 KiB clusters, as QEMU then places the new refcount structures at the
> next refcount block boundary, up to 2 GiB or 8 GiB further.
>
> Existing LVs only get the additional space when they are resized.
>
> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
If we still want to go with this, consider it:
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps
2026-11-13 9:54 ` Fiona Ebner
@ 2026-11-13 11:12 ` Thomas Lamprecht
0 siblings, 0 replies; 8+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 11:12 UTC (permalink / raw)
To: Fiona Ebner, pve-devel
Am 13.11.26 um 10:54 schrieb Fiona Ebner:
> Am 13.11.26 um 2:27 AM schrieb Thomas Lamprecht:
>> Backup solutions like Veeam add persistent dirty bitmaps to qcow2
>> images to track changed blocks. QEMU stores them inside the image when
>
> As others already pointed out, this is not true anymore.
Well, is it fully blocked? As else it can still be possible to do by
other integrator too.
And even if the internal persistent bitmaps are technically somewhat
flawed those can IMO easily look compelling to those – e.g. storage or
backup provider – when adding initial integration.
>> the VM is stopped or migrated, in addition to what 'qemu-img measure'
>> reports and what the LV size is based on. So only the space from
>> rounding up to full extents was left for them, which depends on the
>> image size and can be close to nothing. As QEMU writes all new bitmap
>> clusters before freeing the old ones, a store needs room for two
>> copies, almost 4 MiB for one bitmap on a 900 GiB image. If that space
>> is missing, storing fails with ENOSPC and the bitmap is lost.
>>
>> Reserve space for two copies of two fully dirty bitmaps at the default
>> granularity of 64 KiB, which is also what Veeam was observed to use,
>> plus a refcount block for them. That is about 8 KiB per GiB of disk
>> size, for example 8.4 MiB for 900 GiB. The calculation assumes 128 KiB
>> clusters, so it also covers snapshot overlays. A finer granularity or
>> more than two bitmaps can still exceed the reserved space. So can a
>> store that needs to grow the refcount table, which happens at least for
>> images ending just below 16 TiB with 64 KiB clusters or 128 TiB with
>> 128 KiB clusters, as QEMU then places the new refcount structures at the
>> next refcount block boundary, up to 2 GiB or 8 GiB further.
>>
>> Existing LVs only get the additional space when they are resized.
>>
>> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
> If we still want to go with this, consider it:
>
> Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
It's often just a few MiB that while certainly not a generic safe
guard can protect for a lot of real world pain, I see no big downside
here
^ permalink raw reply [flat|nested] 8+ messages in thread
* applied: [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps
2026-11-13 1:17 ` [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps Thomas Lamprecht
2026-11-13 9:54 ` Fiona Ebner
@ 2026-11-13 12:33 ` Fiona Ebner
1 sibling, 0 replies; 8+ messages in thread
From: Fiona Ebner @ 2026-11-13 12:33 UTC (permalink / raw)
To: pve-devel, Thomas Lamprecht
On Fri, 13 Nov 2026 02:17:25 +0100, Thomas Lamprecht wrote:
> Backup solutions like Veeam add persistent dirty bitmaps to qcow2
> images to track changed blocks. QEMU stores them inside the image when
> the VM is stopped or migrated, in addition to what 'qemu-img measure'
> reports and what the LV size is based on. So only the space from
> rounding up to full extents was left for them, which depends on the
> image size and can be close to nothing. As QEMU writes all new bitmap
> clusters before freeing the old ones, a store needs room for two
> copies, almost 4 MiB for one bitmap on a 900 GiB image. If that space
> is missing, storing fails with ENOSPC and the bitmap is lost.
>
> [...]
Applied this one too now, adapting the commit message slightly to
state that this for Veeam's original integration, thanks!
[3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps
commit: d796f956ac990ded1c1b5e0ca50a451650dc807c
^ permalink raw reply [flat|nested] 8+ messages in thread
* applied: (subset) [PATCH storage 0/3] lvm: fix qcow2 images outgrowing their LV
2026-11-13 1:17 [PATCH storage 0/3] lvm: fix qcow2 images outgrowing their LV Thomas Lamprecht
` (2 preceding siblings ...)
2026-11-13 1:17 ` [PATCH storage 3/3] lvm: qcow2: reserve LV space for persistent dirty bitmaps Thomas Lamprecht
@ 2026-11-13 9:52 ` Fiona Ebner
3 siblings, 0 replies; 8+ messages in thread
From: Fiona Ebner @ 2026-11-13 9:52 UTC (permalink / raw)
To: pve-devel, Thomas Lamprecht
On Fri, 13 Nov 2026 02:17:22 +0100, Thomas Lamprecht wrote:
> There were reports of qcow2 images on LVM storages with
> snapshot-as-volume-chain growing beyond their LV, which pauses the VM
> with an ENOSPC I/O error.
>
> One likely cause is deleting the oldest snapshot while the VM is
> stopped.
> FWICT the plain 'qemu-img commit' used for that drops preallocated
> clusters of the base image wherever the overlay holds zeroes, and
> guest writes to these ranges can later allocate past the end of the LV.
> So the first patch sets 'discard-no-unref' for that commit, as already
> done for running VMs and 'qemu-img convert' in qemu-server's a3a9a2ab
> ("fix #6543: use qcow2 'discard-no-unref' option when using
> snapshot-as-volume-chain").
>
> [...]
Applied the first two patches, thanks!
[1/3] lvm: avoid qcow2 images outgrowing their LV after offline snapshot delete
commit: 01ea0054d5237c61175df767bd400cf832fcf848
[2/3] lvm: qcow2 resize: size the LV for the cluster settings of the image
commit: 00a8186dbefa4b15969da2d395d7e57a0bac7775
^ permalink raw reply [flat|nested] 8+ messages in thread