qemu-server: Severe I/O stalls moving VM disks from NFS to LVM-thin, resolved with -t none

qemu-server: Severe I/O stalls moving VM disks from NFS to LVM-thin, resolved with -t none

[RAFAEL SILVA]

Hello Proxmox VE developers,

I would like to report an issue with VM disk moves from NFS storage to
LVM-thin using the Proxmox VE web interface.

I investigated the issue, identified a difference in the target cache mode
used by qemu-img, and successfully tested a local workaround.

*Environment*

   -

   Proxmox VE: 9.2.0
   -

   pve-manager: 9.2.21
   -

   qemu-server: 9.2.10
   -

   pve-qemu-kvm: 11.0.3-4
   -

   Kernel: 7.0.14-22-pve
   -

   libpve-storage-perl: 9.1.12
   -

   Source: TrueNAS NFS storage containing RAW VM images
   -

   Destination: local LVM-thin storage (local-lvm)
   -

   Hardware: Dell PowerEdge R710, PERC 6/i, four HDDs in RAID 5
   -

   RAID cache policy: WriteThrough (degraded BBU)

*Observed behavior*

Moving a powered-off VM disk from NFS to LVM-thin through the web interface
caused severe I/O congestion and host responsiveness issues.

During the operation, I observed approximately:

   -

   Write latency (w_await): 1,699 ms
   -

   Average queue size (aqu-sz): 104
   -

   Disk utilization: 94.5%
   -

   Write throughput: 10 MB/s

The problem persisted even with a 5 MiB/s conversion bandwidth limit.

It was reproducible with NFS datasets using both 64K and 128K ZFS record
sizes.

Disk moves in the opposite direction (LVM-thin to NFS) worked normally.

VMware ESXi imports into the same LVM-thin storage also worked normally.

*Command generated by Proxmox*

/usr/bin/qemu-img convert -p -n -r 5120K \
  -f raw -O raw -- \
  /mnt/pve/nas-nfs-vms/images/113/vm-113-disk-0.raw \
  zeroinit:/dev/pve/vm-113-disk-0

*Successful manual conversion*

qemu-img convert -p -n -f raw -O raw -t none -- \
  /mnt/pve/nas-nfs-vms/images/113/vm-113-disk-0.raw \
  zeroinit:/dev/pve/vm-113-disk-0

This completed successfully without the previously observed severe I/O
stalls.

I also verified that an ESXi import using -t none opened its destination
block device with O_DIRECT.

*Relevant implementation*

In PVE/QemuServer/QemuImage.pm, the convert() function contains:

push @$cmd, '-t', 'none' if $dst_scfg->{type} eq 'zfspool';

Therefore, the target cache mode is explicitly set to none for ZFS pool
destinations, but not for LVM-thin.

*Locally tested workaround*

I changed the condition to:

push @$cmd, '-t', 'none'
    if $dst_scfg->{type} eq 'zfspool'
    || ($dst_storeid eq 'local-lvm' && $dst_scfg->{type} eq 'lvmthin');

After applying this modification, VM disk moves from NFS to local-lvm
through the Proxmox web interface completed successfully without the
previous severe I/O stalls.

This workaround is intentionally limited to my local storage ID. I am not
proposing the storage-name-specific condition as an upstream solution.

*Additional testing observations*

The original disk move experienced severe I/O stalls even when the
conversion bandwidth was limited to 5 MiB/s.

Both the successful manual conversion and the subsequent successful
web-interface disk move were performed without a bandwidth limit, using -t
none.

Despite the absence of rate limiting, neither operation reproduced the
previously observed I/O congestion.

This suggests that the target cache mode may be a more significant factor
than the transfer rate itself.

However, since the bandwidth limit and target cache mode were changed
together, I cannot conclusively isolate their individual effects.

I have not repeated the failing conversion solely to test this distinction,
as the original behavior caused severe host responsiveness issues.

*Questions*

Would enabling -t none for LVM-thin destinations be appropriate?

Alternatively, would it be preferable to make the target cache mode
configurable for disk conversion operations?

I understand that the degraded RAID BBU and WriteThrough policy may
contribute to the observed behavior, and I cannot yet determine whether the
issue affects other hardware configurations.

I would appreciate feedback on the preferred approach. This would be my
first contribution to Proxmox VE. I would be happy to perform additional
tests, prepare a patch, or follow any guidance on the preferred approach.

Thank you for your time and for maintaining Proxmox VE.
Rafael Prazeres