Re: SPAM: qemu-img convert runs with cache=unsafe for every block storage except zfspool

[Ing. Alfonso Kuen Arroyo <gerencia@idkmanager.com>]

Hi Fiona,

thanks for trying it, and sorry for the slow reply. I re-ran it today on a
PVE 9.2.4 node (pve-qemu-kvm 11.0.2-1, kernel 7.0.14-4-pve) and got the
silent exit 0, so here is the exact recipe, with what I think is the
difference from your runs.

Recipe (loop device on a file, dm-flakey with a 1 s "up" window so the
device fails while the copy is still in flight):

  cd /var/tmp
  head -c 2G /dev/urandom > src.raw          # no zero regions
  truncate -s 2G dst.img
  LOOP=$(losetup --find --show dst.img); SZ=$(blockdev --getsz $LOOP)

  # reset before each run
  dd if=/dev/zero of=$LOOP bs=1M count=2048 conv=fsync status=none
  sync; echo 3 > /proc/sys/vm/drop_caches; dmesg -C
  dmsetup create flk --table "0 $SZ flakey $LOOP 0 1 60"   # 1 s up, 60 s down

  /usr/bin/qemu-img convert -f raw -O raw src.raw /dev/mapper/flk; echo rc=$?
  sleep 70                                   # let the down window and writeback pass
  dmesg | grep -c 'lost async page write'
  dmsetup remove flk
  cmp -l src.raw $LOOP | wc -l

Default cache mode:
  rc=0 (copy took 1.4 s)
  dmesg: "Buffer I/O error on dev dm-0, logical block N, lost async page write"
         (10 lines, then "buffer_io_error: 493941 callbacks suppressed")
  cmp:   2015325567 bytes differ from the source (about 1.9 of 2 GiB never
         made it to the device)

Same run with -t none:
  "qemu-img: error while writing at byte 1763704832: Input/output error"
  rc=1

(qemu-img is the unmodified pve-qemu-kvm binary; strace of that run shows the
target opened O_RDWR without O_DIRECT and a single fdatasync, from the initial
BLKZEROOUT. Our nodes also carry a local wrapper that forces -t none - that is
not what was measured, and the binary name in the -t none error was edited.)

Why I think your runs failed loudly instead: in both of them the device was
already failing at the moment qemu-img wrote to it - dm-error from the start,
and flakey "0 1 1" is down every other second, while mine stays down for 60 s
so the failure lands while the pages are still dirty (a 4 s-up run with a
512 MiB source lost nothing: writeback finished inside the window). With a
qcow2 source I would also expect the holes to go out as write_zeroes, which on
a block device is a synchronous BLKZEROOUT ioctl and fails loudly - but I have
not verified that with a qcow2 source, so treat it as a guess. What I did
verify is that the writes must be accepted into the page cache first and fail
at writeback time, and that the same failure is loud with -t none.

I'll file this on bugzilla.proxmox.com with the recipe and both outputs,
unless you'd prefer to open it on your side.

Best regards,
Alfonso