Re: [RFC PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring

[Dominik Csapak <d.csapak@proxmox.com>]

Hi,

I've read through the cover letter (see my comments inline)
and will go through the patches soon

Overall this will be a good improvement once
it's finished.

On 11/6/26 2:39 PM, Jakob-Tobias Winter wrote:
> Restoring a media set reads one tape after the other with a single
> drive, even if the library has more drives that are idle. This series
> lets a restore use up to three additional drives, so several tapes are
> read at the same time [0]. Even when the host cannot keep up with the
> combined data rate, loading, positioning and changer moves on one
> drive then overlap with reading on the others.
> 
> I am sending this as RFC to discuss the approach.
> 
> The change is kept small and reuses the existing selective restore:
> 
> Patch 1 adds a pool of drives that runs one job per media. With a
> single drive, this is the existing loop, so a restore with one drive
> behaves as before. With several drives, each drive runs in its own
> thread and takes the media already loaded in it, else the next one
> that is not loaded in another drive of the pool. Changer operations
> are serialized, so two drives do not pick the same free slot when
> unloading. The first error, or an abort, stops all drives. As the lines
> of all drives are interleaved in the task log, every message of a drive
> starts with "drive 'NAME': ".
> 
> Patch 2 loads the media of the first two phases of the selective
> restore through the pool, without functional change.
> 
> Patch 3 adds the optional 'additional-drives' parameter to the API and
> to 'proxmox-tape restore'. Every drive is checked for Tape.Read, locked
> and marked as in use for the whole task. A selective restore then reads
> the snapshot archives (phase 1) and the chunks (phase 2) on all drives.
> 
> A full restore writes the snapshot archives straight into the
> datastore. That is safe with a single drive, because a tape backup
> writes the chunks of a snapshot before its snapshot archive, to the
> same or an earlier media. With several drives, a snapshot archive can
> be read while an earlier media is still being restored, and an aborted
> restore would leave snapshots with missing chunks behind. So with more
> than one drive, a full restore runs the phases of the selective restore
> for all snapshots, which only add the snapshots once all chunks are
> restored. The restore window already takes that path whenever it sends
> a namespace mapping. The trade-offs are that the media set catalog is
> needed, every media is loaded twice (once for the snapshot archives,
> once for the chunks), and the snapshots only show up at the end.
> 

maybe the double loading of tapes could be avoided by restoring
the snapshots of the subsequent tapes in a temporary location
and moving those afterwards? the first tape is generally
exempt here since there the guarantees must hold anyway
(but special casing this might be overkill)

so we could simply go through each tape, restore all chunks
to the chunks dir and snapshots to a tmpdir, then no tape
has to be loaded twice?

> Patch 4 adds a multi-select field for the additional drives to the
> restore window.
> 
> On abort or error, the existing cleanup of the selective restore
> removes the temporary directory, so an abort before phase 3 does not
> add any snapshot. Chunks that were already written stay until garbage
> collection removes them, as today.
> 
> Open questions:
> 
> - Is 'drive' plus 'additional-drives' fine, or would you prefer
>    something else, e.g. a 'drives' list, or the changer plus "use all
>    free drives"?

i'd probably add a 'drives' parameter that replaces the 'drive' one.
we could then always use that and deprecate 'drive'

this is the most flexible and straightforward way i think.


> - Is a maximum of four drives in total OK or can we drop the limit?

I'd probably not restrict it to an arbitrary limit, since it's
impossible to know what is a good limit beforehand.
The number of drives directly controls the number of threads and 
necessary read performance, so the user has to know what his system
can handle.

> - Standalone drives (without changer) work, but the pool cannot know
>    which tape is in which drive, so the operator has to follow the
>    per-drive "insert media" notifications. Is that OK?

Not sure this feature makes too much sense for standalone drives
at all, since it's rather unlikely that a single pbs install
has multiple standalone drives intended for a single restore
job, but yes, following the notification mails is ok IMO.

> - Should the "drive 'NAME': " prefix also be used with a single drive,
>    for consistency?

no, in that case i would simply write the used drive at the beginning of
the log somewhere so it's clear (when not done already).

> 
> Testing: I built the series on top of 4.2.8, and tested it on a tape
> library with four drives (LTO-7 media), with one and with several
> drives. With four drives, each drive read at about 260-290 MB/s.
> 
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7333
> 
> Jakob-Tobias Winter (4):
>    tape: drive: add a pool to use several drives for one task
>    tape: restore: use a drive pool for the selective restore
>    close #7333: tape: restore: allow using several drives in parallel
>    ui: tape restore: add selector for additional drives
> 
>   src/api2/tape/restore.rs       | 179 +++++++++++------
>   src/bin/proxmox-tape.rs        |  10 +
>   src/tape/drive/mod.rs          |  41 +++-
>   src/tape/drive/pool.rs         | 358 +++++++++++++++++++++++++++++++++
>   www/tape/window/TapeRestore.js |  15 ++
>   5 files changed, 540 insertions(+), 63 deletions(-)
>   create mode 100644 src/tape/drive/pool.rs
> 
> 
> base-commit: 1df6dff9c0acdd11386245502b2c7aeed2f440dd