[RFC PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring

[Jakob-Tobias Winter <tobias@stuttgart.de>]

Restoring a media set reads one tape after the other with a single
drive, even if the library has more drives that are idle. This series
lets a restore use up to three additional drives, so several tapes are
read at the same time [0]. Even when the host cannot keep up with the
combined data rate, loading, positioning and changer moves on one
drive then overlap with reading on the others.

I am sending this as RFC to discuss the approach.

The change is kept small and reuses the existing selective restore:

Patch 1 adds a pool of drives that runs one job per media. With a
single drive, this is the existing loop, so a restore with one drive
behaves as before. With several drives, each drive runs in its own
thread and takes the media already loaded in it, else the next one
that is not loaded in another drive of the pool. Changer operations
are serialized, so two drives do not pick the same free slot when
unloading. The first error, or an abort, stops all drives. As the lines
of all drives are interleaved in the task log, every message of a drive
starts with "drive 'NAME': ".

Patch 2 loads the media of the first two phases of the selective
restore through the pool, without functional change.

Patch 3 adds the optional 'additional-drives' parameter to the API and
to 'proxmox-tape restore'. Every drive is checked for Tape.Read, locked
and marked as in use for the whole task. A selective restore then reads
the snapshot archives (phase 1) and the chunks (phase 2) on all drives.

A full restore writes the snapshot archives straight into the
datastore. That is safe with a single drive, because a tape backup
writes the chunks of a snapshot before its snapshot archive, to the
same or an earlier media. With several drives, a snapshot archive can
be read while an earlier media is still being restored, and an aborted
restore would leave snapshots with missing chunks behind. So with more
than one drive, a full restore runs the phases of the selective restore
for all snapshots, which only add the snapshots once all chunks are
restored. The restore window already takes that path whenever it sends
a namespace mapping. The trade-offs are that the media set catalog is
needed, every media is loaded twice (once for the snapshot archives,
once for the chunks), and the snapshots only show up at the end.

Patch 4 adds a multi-select field for the additional drives to the
restore window.

On abort or error, the existing cleanup of the selective restore
removes the temporary directory, so an abort before phase 3 does not
add any snapshot. Chunks that were already written stay until garbage
collection removes them, as today.

Open questions:

- Is 'drive' plus 'additional-drives' fine, or would you prefer
  something else, e.g. a 'drives' list, or the changer plus "use all
  free drives"?
- Is a maximum of four drives in total OK or can we drop the limit?
- Standalone drives (without changer) work, but the pool cannot know
  which tape is in which drive, so the operator has to follow the
  per-drive "insert media" notifications. Is that OK?
- Should the "drive 'NAME': " prefix also be used with a single drive,
  for consistency?

Testing: I built the series on top of 4.2.8, and tested it on a tape
library with four drives (LTO-7 media), with one and with several
drives. With four drives, each drive read at about 260-290 MB/s.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=7333

Jakob-Tobias Winter (4):
  tape: drive: add a pool to use several drives for one task
  tape: restore: use a drive pool for the selective restore
  close #7333: tape: restore: allow using several drives in parallel
  ui: tape restore: add selector for additional drives

 src/api2/tape/restore.rs       | 179 +++++++++++------
 src/bin/proxmox-tape.rs        |  10 +
 src/tape/drive/mod.rs          |  41 +++-
 src/tape/drive/pool.rs         | 358 +++++++++++++++++++++++++++++++++
 www/tape/window/TapeRestore.js |  15 ++
 5 files changed, 540 insertions(+), 63 deletions(-)
 create mode 100644 src/tape/drive/pool.rs


base-commit: 1df6dff9c0acdd11386245502b2c7aeed2f440dd
-- 
2.53.0