[PATCH docs/manager/qemu-server v3 0/6] partially fix #1989: add QEMU qcow2 cache options

[Erik Fastermann <e.fastermann@proxmox.com>]

Add multiple options to configure the qcow2 L2/refcount cache. This
can provide significant performance gains in some cases, mainly for
large disks.

The pve-manager patch requires a qemu-server version that includes
patch 4, since older versions reject the new qcow2-* drive properties.
A v2 of the manager patch was applied and reverted (43b51f49) for that
reason.

Patches 1-3 are new and do not depend on the cache options:

1: re-run proxmox-perltidy, which reformats two calls in QemuServer.pm
   on current master. No functional change.
2: fix the bps, bps_rd and bps_wr throttle options, which were
   converted to 0 mbps (unlimited) instead of their actual value.
3: make the API report why parse_drive rejects a drive. Previously,
   the web UI only showed "unable to parse drive options" when one of
   the new qcow2 checks, or one of the existing checks, failed.

Changes since v2:

qemu-server (patch 4):
* Legacy -drive commandline (machine version < 10.0): prefix the
  options with "file." during a live restore, so they are set on the
  qcow2 node. Previously they ended up on the alloc-track node, which
  rejects them, so the live restore could not start.
* Handle the undefined format of empty CD-ROM drives. This caused an
  uninitialized value warning on every start and broke some cfg2cmd
  tests.
* Automatic sizing uses the virtual size reported by qemu-img info
  instead of the optional size= property. Without size=, the cache
  fell back to its minimum regardless of the disk size.
* Call qemu-img info with the qcow2 format and a 10 second timeout.
* Limit qcow2-cache-clean-interval to 2^32 - 1, since QEMU refuses to
  start with larger values, and describe the option in more detail.
* Report validation errors via the mechanism from patch 3 and use the
  full property names in the messages.
* The QEMU default L2 cache is up to 32 MiB, not always 32 MiB.
* Drop an unused log_warn import from Blockdev.pm.
* Rebase onto current master.

pve-manager (patch 5):
* Keep the underscore in the drive parser's key regex. Without it,
  disks with throttling options (mbps_rd, iops_wr_max, ...) could no
  longer be edited.
* Enabling automatic sizing removes qcow2-l2-cache-size and
  qcow2-refcount-cache-size if they were set via the API or CLI, since
  the backend rejects that combination.
* Parse the automatic sizing flag with PVE.Parser.parseBoolean like
  the neighboring fields.
* Tooltip: the QEMU default L2 cache is up to 32 MiB.
* Only accept whole numbers up to 10240 in the qcow2 cache size field,
  matching what the backend accepts.

pve-docs (patch 6):
* Name the options that are not exposed in the web UI and mention
  qm set.
* Note that QEMU allocates the cache in addition to the guest memory.
* Mention that the options mainly help with large disks.

Known limitations:

* Snapshots as volume chain: every qcow2 node of the backing chain
  gets the same cache options. With automatic sizing, every layer gets
  the cache size computed for the whole disk, so the worst-case memory
  use grows with the number of snapshots (for a 2 TiB disk with three
  snapshots, 4 x 320 MiB instead of 320 MiB), and qemu-img info runs
  once per layer. I would be inclined to leave it this way for now.

* Moving a disk to a storage that does not use qcow2 keeps the qcow2-*
  options, so the move and every later VM start log a warning for each
  of them. The options could be dropped silently on move instead, but
  the warning is also useful to have. The downside is that the warning
  cannot be cleared from the web UI, since the qcow2 fields are
  disabled for non-qcow2 disks, so the options can only be removed via
  the API or CLI.

* The disk dialog only exposes the total cache size. If
  qcow2-l2-cache-size or qcow2-refcount-cache-size were set via the
  API or CLI and are not smaller than the entered total, or both are
  set, saving fails. I think this is acceptable for now, and with patch
  3 the error message states which options conflict.

* With patch 3, problems for which parse_drive previously returned
  undef silently are now logged as warnings for callers that do not
  collect the errors. This includes the early returns for an invalid
  drive key and for a drive string that cannot be parsed. The warning
  adds useful information, so I would keep it.

Thanks to Fiona and Dominik for the feedback on the previous versions!


qemu-server:

Erik Fastermann (4):
  re-tidy source code
  drive: fix conversion of bps throttle limits to mbps
  api: report why drive options are rejected
  partially fix #1989: disk: add qcow2 cache options

 src/PVE/API2/Qemu.pm           |   9 +-
 src/PVE/QemuServer.pm          |  47 +++++-
 src/PVE/QemuServer/Blockdev.pm |   7 +
 src/PVE/QemuServer/Drive.pm    | 263 ++++++++++++++++++++++++++++++---
 4 files changed, 302 insertions(+), 24 deletions(-)


pve-manager:

Erik Fastermann (1):
  partially fix #1989: ui: qemu: disk: add qcow2 cache size config

 www/manager6/Parser.js      |  2 +-
 www/manager6/qemu/HDEdit.js | 81 +++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+), 1 deletion(-)


pve-docs:

Erik Fastermann (1):
  partially fix #1989: qm: document qcow2 cache size config

 qm.adoc | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)


Summary over all repositories:
  7 files changed, 400 insertions(+), 25 deletions(-)

-- 
Generated by murpp 0.12.1