* [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring
@ 2026-11-12 11:37 Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 1/4] tape: drive: add a pool to use several drives for one task Jakob-Tobias Winter
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Jakob-Tobias Winter @ 2026-11-12 11:37 UTC (permalink / raw)
To: pbs-devel; +Cc: Jakob-Tobias Winter
Restoring a media set reads one tape after the other with a single
drive, even if the library has more drives that are idle. This series
lets a restore use several drives of a changer, so several tapes are
read at the same time [0]. Even when the host cannot keep up with the
combined data rate, loading, positioning and changer moves on one
drive then overlap with reading on the others.
This is the follow-up to the RFC [1], thanks to Dominik for the
review.
Patch 1 adds a pool of drives that runs one job per media. With a
single drive, this is the existing loop, so a restore with one drive
behaves as before. With several drives, the media in the drives are
unloaded first, and the task fails if a needed media is loaded in a
drive that is not used. Then each drive runs in its own thread, where
it takes the next media, loads it, runs the job and unloads it again.
Changer operations are serialized, so two drives do not pick the same
free slot when unloading. The first error, or an abort, stops all
drives, which then unload their media as well. As the lines of all
drives are interleaved in the task log, every message of a drive
starts with "drive 'NAME': ".
Patch 2 loads the media of the first two phases of the selective
restore through the pool, without functional change.
Patch 3 adds the 'drives' parameter to the API and to 'proxmox-tape
restore'. It replaces 'drive', which still works, but is deprecated.
Several drives have to be in the same changer. Every drive is checked
for Tape.Read, locked and marked as in use for the whole task. A
selective restore then reads the snapshot archives (phase 1) and the
chunks (phase 2) on all drives.
A full restore writes the snapshot archives straight into the
datastore. That is safe with a single drive, because a tape backup
writes the chunks of a snapshot before its snapshot archive, to the
same or an earlier media. With several drives, a snapshot archive can
be read while an earlier media is still being restored, and an aborted
restore would leave snapshots with missing chunks behind. So with more
than one drive, a full restore runs the phases of the selective restore
for all snapshots, which only add the snapshots once all chunks are
restored. The restore window already takes that path whenever it sends
a namespace mapping. The trade-offs are that the media set catalog is
needed, every media is loaded twice (once for the snapshot archives,
once for the chunks), the snapshots only show up at the end, and
snapshots that already exist in the datastore are skipped with a
warning instead of silently.
Restoring everything in a single pass, with the snapshots in a
temporary directory until all chunks are there, would avoid the second
load, as suggested in the review of the RFC. I left that out to keep
the changes to the restore logic small, and would look into it as a
follow-up.
Patch 4 turns the drive selector of the restore window into a
multi-select field, which only accepts several drives if they are in
the same changer.
On abort or error, the existing cleanup of the selective restore
removes the temporary directory, so an abort before phase 3 does not
add any snapshot. Chunks that were already written stay until garbage
collection removes them, as today.
Changes since the RFC [1]:
- 'drives' replaces 'drive', instead of adding 'additional-drives',
and there is no limit for the number of drives anymore
- several drives have to be in the same changer, the API and the
restore window check that
- the drives are unloaded first, a needed media in a drive that is not
used is an error, and each drive takes the next media in order and
unloads it after use, instead of preferring the media loaded in it
- a failing drive aborts the task, instead of using a stop flag and a
task context of its own, and a poisoned lock is not ignored anymore
- the pool takes the maps of the two restore phases as they are and
looks up the media labels in the inventory itself
- one multi-select field in the restore window instead of a second
field for additional drives
- rebased on a newer master
Testing: I built this version on top of 4.2.8 and tested it with
several drives on a tape library with LTO-7 media, where four of the
drives are available to PBS: restoring a whole media set, restoring a
selected snapshot, aborting a restore, after which the drives unload
their media, and the error for a needed media that is loaded in a
drive that is not used. I also tested the unloading of drives that
hold a media when the restore starts, and the restore window.
In addition, with real daemons and four virtual tape drives, each with
a copy of the tapes: with one drive, the task logs of a full restore,
a restore of selected snapshots and one with a namespace mapping are
the same as on master. With two to four drives, the same three kinds
of restore give datastores with the content of the source, which also
verify and read back correctly. A drive that fails stops the others
and unloads its media, and the task ends with its error, without
adding a snapshot. Drives that are not in the same changer are
refused. The test suite passes and clippy shows no new warnings.
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=7333
[1] https://lore.proxmox.com/pbs-devel/20261106133826.1692620-2-tobias@stuttgart.de/
Jakob-Tobias Winter (4):
tape: drive: add a pool to use several drives for one task
tape: restore: use a drive pool for the selective restore
close #7333: tape: restore: allow using several drives in parallel
ui: tape restore: allow selecting several drives
src/api2/tape/restore.rs | 180 ++++++++++++++++-------
src/bin/proxmox-tape.rs | 15 +-
src/tape/drive/mod.rs | 40 ++++-
src/tape/drive/pool.rs | 259 +++++++++++++++++++++++++++++++++
www/tape/form/DriveSelector.js | 17 +++
www/tape/window/TapeRestore.js | 5 +-
6 files changed, 455 insertions(+), 61 deletions(-)
create mode 100644 src/tape/drive/pool.rs
base-commit: 68577e65400958d9f7e76a7722669fb359b69d4b
--
2.53.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH proxmox-backup 1/4] tape: drive: add a pool to use several drives for one task
2026-11-12 11:37 [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring Jakob-Tobias Winter
@ 2026-11-12 11:37 ` Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 2/4] tape: restore: use a drive pool for the selective restore Jakob-Tobias Winter
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Jakob-Tobias Winter @ 2026-11-12 11:37 UTC (permalink / raw)
To: pbs-devel; +Cc: Jakob-Tobias Winter
Restoring a media set loads one tape after the other into a single
drive, even if the library has more drives that are idle. In
preparation for using them in parallel, add a pool of drives that runs
a list of jobs, each needing one media, on all of its drives.
With a single drive, the jobs run in the given order in the calling
thread, exactly like the existing loops that load one media after the
other.
With several drives, the media in the drives of the pool are unloaded
first. If a needed media is loaded in another drive, it cannot be
loaded, so the task fails right away. Then each drive runs in its own
thread, where it takes the next media in order, loads it, runs the job
and unloads it again, also if the job failed or was aborted. Changer
operations are serialized, so two drives do not pick the same free
slot when unloading a media. A drive rewinds and ejects its media
before that, so the others only wait while the changer moves a media.
The first error aborts the task, which stops the other drives, and is
returned once all of them finished.
As the lines of all drives are interleaved in the task log, their
messages start with the name of the drive, and the jobs get this
prefix for their own messages. With a single drive, the prefix is
empty, so the log stays as it is.
Suggested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
---
changes since the RFC:
- each drive takes the next media in order instead of preferring the
one that is loaded in it: all drives are unloaded first, and the
task fails if a needed media is loaded in a drive that is not used
- a drive unloads its media after the job, also if the job failed or
was aborted, and ejects it before it waits for the changer
- a failing drive aborts the task instead of setting a stop flag of
its own, so the separate task context and the panic guard are gone
- panic on a poisoned lock instead of ignoring it
- the jobs are a map with the media uuids as keys, and the pool looks
up the labels in the inventory
- the pool keeps the drive and hands it to the job by reference, to
eject the media afterwards
src/tape/drive/mod.rs | 40 ++++++-
src/tape/drive/pool.rs | 259 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 293 insertions(+), 6 deletions(-)
create mode 100644 src/tape/drive/pool.rs
diff --git a/src/tape/drive/mod.rs b/src/tape/drive/mod.rs
index fa84fd54a..45f75b768 100644
--- a/src/tape/drive/mod.rs
+++ b/src/tape/drive/mod.rs
@@ -1,6 +1,7 @@
//! Tape drivers
use std::path::PathBuf;
+use std::sync::Mutex;
use anyhow::{Error, bail, format_err};
use nix::fcntl::OFlag;
@@ -39,6 +40,9 @@ mod virtual_tape;
mod lto;
pub use lto::*;
+mod pool;
+pub use pool::*;
+
/// Tape driver interface
pub trait TapeDriver {
/// Flush all data to the tape
@@ -378,11 +382,28 @@ pub fn request_and_load_media(
drive: &str,
label: &MediaLabel,
notification_mode: &TapeNotificationMode,
+) -> Result<(Box<dyn TapeDriver>, MediaId), Error> {
+ request_and_load_media_locked(worker, config, drive, label, notification_mode, None, "")
+}
+
+/// Like [`request_and_load_media`], but holds `changer_lock` while the changer moves media and
+/// starts each log message with `log_prefix`.
+///
+/// This serializes the changer operations of several drives used by the same task, and tells
+/// their messages apart in the task log.
+pub fn request_and_load_media_locked(
+ worker: &dyn WorkerTaskContext,
+ config: &SectionConfigData,
+ drive: &str,
+ label: &MediaLabel,
+ notification_mode: &TapeNotificationMode,
+ changer_lock: Option<&Mutex<()>>,
+ log_prefix: &str,
) -> Result<(Box<dyn TapeDriver>, MediaId), Error> {
let check_label = |handle: &mut dyn TapeDriver, uuid: &proxmox_uuid::Uuid| {
if let Ok((Some(media_id), _)) = handle.read_label() {
info!(
- "found media label {} ({})",
+ "{log_prefix}found media label {} ({})",
media_id.label.label_text, media_id.label.uuid,
);
@@ -422,7 +443,7 @@ pub fn request_and_load_media(
new: TapeRequestError|
-> Result<(), Error> {
if new != *old {
- info!("{new}");
+ info!("{log_prefix}{new}");
let (device_type, device) = if let Some(changer) = changer {
("changer", changer.as_str())
} else {
@@ -430,7 +451,8 @@ pub fn request_and_load_media(
};
info!(
- "Please insert media '{label_text}' into {device_type} '{device}'"
+ "{log_prefix}Please insert media '{label_text}' into {device_type} \
+ '{device}'"
);
send_load_media_notification(
notification_mode,
@@ -455,13 +477,19 @@ pub fn request_and_load_media(
std::thread::sleep(std::time::Duration::from_millis(100));
}
} else if drive_config.changer.is_none() {
- info!("Checking for media '{label_text}' in drive '{drive}'");
+ info!(
+ "{log_prefix}Checking for media '{label_text}' in drive '{drive}'"
+ );
} else {
- info!("trying to load media '{label_text}' into drive '{drive}'");
+ info!(
+ "{log_prefix}trying to load media '{label_text}' into drive \
+ '{drive}'"
+ );
}
if drive_config.changer.is_some() {
let mut changer = MtxMediaChanger::with_drive_config(&drive_config)?;
+ let _changer_guard = changer_lock.map(|lock| lock.lock().unwrap());
if let Err(err) = changer.load_media(&label_text) {
update_and_log_request_error(
&mut last_error,
@@ -485,7 +513,7 @@ pub fn request_and_load_media(
let request_error = match handle.read_label() {
Ok((Some(media_id), _)) if media_id.label.uuid == label.uuid => {
info!(
- "found media label {} ({})",
+ "{log_prefix}found media label {} ({})",
media_id.label.label_text,
media_id.label.uuid.to_string(),
);
diff --git a/src/tape/drive/pool.rs b/src/tape/drive/pool.rs
new file mode 100644
index 000000000..3906769e5
--- /dev/null
+++ b/src/tape/drive/pool.rs
@@ -0,0 +1,259 @@
+//! Use several tape drives for one task
+
+use std::collections::{BTreeMap, HashMap, HashSet};
+use std::sync::{Arc, Mutex};
+
+use anyhow::{Error, bail};
+use tracing::{info, warn};
+
+use proxmox_log::LogContext;
+use proxmox_rest_server::WorkerTask;
+use proxmox_section_config::SectionConfigData;
+use proxmox_uuid::Uuid;
+use proxmox_worker_task::WorkerTaskContext;
+
+use pbs_tape::ElementStatus;
+
+use crate::tape::{Inventory, MediaId, TapeNotificationMode, file_formats::MediaLabel};
+
+use super::{TapeDriver, media_changer, request_and_load_media, request_and_load_media_locked};
+
+/// A media loaded into a drive of a [`DrivePool`].
+pub struct LoadedMedia {
+ pub drive: Box<dyn TapeDriver>,
+ pub media_id: MediaId,
+ /// The start of the job's log messages, names the drive if the pool has several.
+ pub log_prefix: String,
+}
+
+/// A set of drives that work on the media of one task in parallel.
+pub struct DrivePool<'a> {
+ worker: Arc<WorkerTask>,
+ config: &'a SectionConfigData,
+ inventory: &'a Inventory,
+ drives: &'a [String],
+ notification_mode: &'a TapeNotificationMode,
+ /// Serializes the changer operations of all drives.
+ changer_lock: Mutex<()>,
+}
+
+impl<'a> DrivePool<'a> {
+ /// Create a pool of `drives`, which the caller has to lock.
+ pub fn new(
+ worker: Arc<WorkerTask>,
+ config: &'a SectionConfigData,
+ inventory: &'a Inventory,
+ drives: &'a [String],
+ notification_mode: &'a TapeNotificationMode,
+ ) -> Self {
+ Self {
+ worker,
+ config,
+ inventory,
+ drives,
+ notification_mode,
+ changer_lock: Mutex::new(()),
+ }
+ }
+
+ /// Load each media of `jobs` into a drive and run `func` with its job.
+ ///
+ /// With a single drive, the jobs run in the order of the map in the calling thread.
+ ///
+ /// With several drives, the media in the drives are unloaded first, and a media of a job that
+ /// is loaded in another drive is an error. Then each drive runs in its own thread, where it
+ /// takes the first job that is left, loads its media, runs `func` and unloads the media
+ /// again, also if `func` failed. The first error aborts the task, which stops the jobs on
+ /// all drives, and is returned once all of them finished.
+ ///
+ /// The results are returned in the order the jobs finished.
+ pub fn run<T, R, F>(&self, jobs: BTreeMap<Uuid, T>, func: F) -> Result<Vec<R>, Error>
+ where
+ T: Send,
+ R: Send,
+ F: Fn(&mut LoadedMedia, T) -> Result<R, Error> + Sync,
+ {
+ if let [drive_name] = self.drives {
+ let mut results = Vec::with_capacity(jobs.len());
+ for (media_uuid, job) in jobs {
+ let (drive, media_id) = request_and_load_media(
+ &*self.worker,
+ self.config,
+ drive_name,
+ self.label(&media_uuid)?,
+ self.notification_mode,
+ )?;
+ let mut media = LoadedMedia {
+ drive,
+ media_id,
+ log_prefix: String::new(),
+ };
+ results.push(func(&mut media, job)?);
+ }
+ return Ok(results);
+ }
+
+ self.unload_drives(&jobs)?;
+
+ let queue = Mutex::new(jobs);
+ let results = Mutex::new(Vec::new());
+ let first_error = Mutex::new(None);
+ let log_context = LogContext::current();
+
+ std::thread::scope(|scope| {
+ for drive_name in self.drives {
+ let (queue, results, first_error, func) = (&queue, &results, &first_error, &func);
+ let log_context = log_context.clone();
+
+ scope.spawn(move || {
+ let run = || {
+ let mut loaded = None;
+ let res = self.drive_loop(drive_name, queue, results, func, &mut loaded);
+ if let Err(err) = res {
+ let mut first_error = first_error.lock().unwrap();
+ if first_error.is_none() {
+ if !self.worker.abort_requested() {
+ warn!(
+ "drive '{drive_name}': failed, stopping all drives - {err}"
+ );
+ self.worker.request_abort();
+ }
+ *first_error = Some(err);
+ }
+ }
+ // a job that failed or was aborted left its media in the drive
+ let unloaded = loaded.map(|media| self.unload_media(drive_name, media));
+ if let Some(Err(err)) = unloaded {
+ warn!("drive '{drive_name}': unable to unload media - {err}");
+ }
+ };
+ match log_context {
+ Some(log_context) => log_context.sync_scope(run),
+ None => run(),
+ }
+ });
+ }
+ });
+
+ if let Some(err) = first_error.into_inner().unwrap() {
+ return Err(err);
+ }
+
+ Ok(results.into_inner().unwrap())
+ }
+
+ fn label(&self, media_uuid: &Uuid) -> Result<&'a MediaLabel, Error> {
+ match self.inventory.lookup_media(media_uuid) {
+ Some(media_id) => Ok(&media_id.label),
+ None => bail!("unable to lookup media {media_uuid}"),
+ }
+ }
+
+ /// Unload the media of all drives, then make sure that no media of `jobs` is loaded in a
+ /// drive that is not part of this pool, as it cannot be loaded from there.
+ ///
+ /// Drives without a changer are skipped.
+ fn unload_drives<T>(&self, jobs: &BTreeMap<Uuid, T>) -> Result<(), Error> {
+ let mut changers = HashMap::new();
+
+ for drive_name in self.drives {
+ let Some((mut changer, changer_name)) = media_changer(self.config, drive_name)? else {
+ continue;
+ };
+ let mut status = changer.status()?;
+ let loaded = status
+ .drives
+ .get(changer.drive_number() as usize)
+ .is_some_and(|drive| !matches!(drive.status, ElementStatus::Empty));
+ if loaded {
+ info!("drive '{drive_name}': unloading media");
+ status = changer.unload_media(None)?;
+ }
+ changers.insert(changer_name, status);
+ }
+
+ let mut needed = HashSet::new();
+ for media_uuid in jobs.keys() {
+ needed.insert(self.label(media_uuid)?.label_text.as_str());
+ }
+
+ for (changer_name, status) in changers {
+ for (drive_number, drive) in status.drives.iter().enumerate() {
+ let ElementStatus::VolumeTag(ref tag) = drive.status else {
+ continue;
+ };
+ if needed.contains(tag.as_str()) {
+ bail!(
+ "media '{tag}' is loaded in drive {drive_number} of changer \
+ '{changer_name}', which this task does not use"
+ );
+ }
+ }
+ }
+
+ Ok(())
+ }
+
+ fn drive_loop<T, R, F>(
+ &self,
+ drive_name: &str,
+ queue: &Mutex<BTreeMap<Uuid, T>>,
+ results: &Mutex<Vec<R>>,
+ func: &F,
+ loaded: &mut Option<LoadedMedia>,
+ ) -> Result<(), Error>
+ where
+ F: Fn(&mut LoadedMedia, T) -> Result<R, Error>,
+ {
+ // the lines of all drives are interleaved in the task log
+ let log_prefix = format!("drive '{drive_name}': ");
+
+ loop {
+ self.worker.check_abort()?;
+
+ let Some((media_uuid, job)) = queue.lock().unwrap().pop_first() else {
+ return Ok(());
+ };
+ let label = self.label(&media_uuid)?;
+
+ let (drive, media_id) = request_and_load_media_locked(
+ &*self.worker,
+ self.config,
+ drive_name,
+ label,
+ self.notification_mode,
+ Some(&self.changer_lock),
+ &log_prefix,
+ )?;
+ // stays in `loaded` if the job fails, the caller then unloads the media
+ let media = loaded.insert(LoadedMedia {
+ drive,
+ media_id,
+ log_prefix: log_prefix.clone(),
+ });
+ let result = func(media, job)?;
+ results.lock().unwrap().push(result);
+
+ if let Some(media) = loaded.take() {
+ self.unload_media(drive_name, media)?;
+ }
+ }
+ }
+
+ /// Unload the media from a drive after its job, if the drive is in a changer.
+ fn unload_media(&self, drive_name: &str, mut media: LoadedMedia) -> Result<(), Error> {
+ let Some((mut changer, _)) = media_changer(self.config, drive_name)? else {
+ return Ok(());
+ };
+ info!(
+ "{}unloading media '{}'",
+ media.log_prefix, media.media_id.label.label_text
+ );
+ // rewind and eject first, so that the other drives only wait for the changer
+ media.drive.eject_media()?;
+ drop(media);
+ let _changer_guard = self.changer_lock.lock().unwrap();
+ changer.unload_media(None)?;
+ Ok(())
+ }
+}
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH proxmox-backup 2/4] tape: restore: use a drive pool for the selective restore
2026-11-12 11:37 [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 1/4] tape: drive: add a pool to use several drives for one task Jakob-Tobias Winter
@ 2026-11-12 11:37 ` Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 3/4] close #7333: tape: restore: allow using several drives in parallel Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 4/4] ui: tape restore: allow selecting several drives Jakob-Tobias Winter
3 siblings, 0 replies; 5+ messages in thread
From: Jakob-Tobias Winter @ 2026-11-12 11:37 UTC (permalink / raw)
To: pbs-devel; +Cc: Jakob-Tobias Winter
In preparation for restoring from several drives in parallel, load the
media of the first two phases of a selective restore through a drive
pool, and start the messages logged while reading a media with the
prefix the pool provides. The pool only contains the given drive for
now, so the media are loaded in the same order as before and the
prefix is empty.
No functional change intended.
Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
---
changes since the RFC:
- both phases hand their maps to the pool as they are, instead of
turning them into lists with the media labels first
- the restore functions keep taking the task itself, as the pool has
no task context of its own anymore
- the snapshot restore borrows the drive instead of taking it over
src/api2/tape/restore.rs | 99 ++++++++++++++++++++++++----------------
1 file changed, 59 insertions(+), 40 deletions(-)
diff --git a/src/api2/tape/restore.rs b/src/api2/tape/restore.rs
index 6544aa726..9f0111216 100644
--- a/src/api2/tape/restore.rs
+++ b/src/api2/tape/restore.rs
@@ -40,7 +40,9 @@ use pbs_tape::{
use crate::backup::check_ns_modification_privs;
use crate::tape::{
Inventory, MediaCatalog, MediaId, MediaSet, MediaSetCatalog, TAPE_STATUS_DIR,
- drive::{TapeDriver, lock_tape_device, request_and_load_media, set_tape_device_state},
+ drive::{
+ DrivePool, TapeDriver, lock_tape_device, request_and_load_media, set_tape_device_state,
+ },
file_formats::{
CatalogArchiveHeader, ChunkArchiveDecoder, ChunkArchiveHeader,
PROXMOX_BACKUP_CATALOG_ARCHIVE_MAGIC_1_0, PROXMOX_BACKUP_CATALOG_ARCHIVE_MAGIC_1_1,
@@ -418,7 +420,7 @@ pub fn restore(
inventory,
media_set_uuid,
drive_config,
- &drive,
+ std::slice::from_ref(&drive),
store_map,
restore_owner,
¬ification_mode,
@@ -628,7 +630,7 @@ fn restore_list_worker(
inventory: Inventory,
media_set_uuid: Uuid,
drive_config: SectionConfigData,
- drive_name: &str,
+ drives: &[String],
store_map: DataStoreMap,
restore_owner: &Authid,
notification_mode: &TapeNotificationMode,
@@ -636,6 +638,13 @@ fn restore_list_worker(
auth_id: &Authid,
) -> Result<(), Error> {
let catalog = get_media_set_catalog(&inventory, &media_set_uuid)?;
+ let drive_pool = DrivePool::new(
+ worker.clone(),
+ &drive_config,
+ &inventory,
+ drives,
+ notification_mode,
+ );
let mut datastore_locks = Vec::new();
let mut snapshot_file_hash: BTreeMap<Uuid, Vec<(u64, String)>> = BTreeMap::new();
@@ -760,30 +769,32 @@ fn restore_list_worker(
info!("Phase 1: temporarily restore snapshots to temp dir");
log_required_tapes(&inventory, snapshot_file_hash.keys());
- let mut datastore_chunk_map: HashMap<String, HashSet<[u8; 32]>> = HashMap::new();
- let mut tmp_paths = Vec::new();
- for (media_uuid, file_list) in snapshot_file_hash.iter_mut() {
- let media_id = inventory.lookup_media(media_uuid).unwrap();
- let (drive, info) = request_and_load_media(
- &worker,
- &drive_config,
- drive_name,
- &media_id.label,
- notification_mode,
- )?;
+ let results = drive_pool.run(snapshot_file_hash, |media, mut file_list| {
file_list.sort_unstable();
-
+ let mut chunk_map = HashMap::new();
let tmp_path = restore_snapshots_to_tmpdir(
worker.clone(),
&store_map,
- file_list,
- drive,
- &info,
+ &file_list,
+ &mut media.drive,
+ &media.media_id,
&media_set_uuid,
- &mut datastore_chunk_map,
+ &mut chunk_map,
+ &media.log_prefix,
)
.map_err(|err| format_err!("could not restore snapshots to tmpdir: {}", err))?;
+ Ok((tmp_path, chunk_map))
+ })?;
+ let mut datastore_chunk_map: HashMap<String, HashSet<[u8; 32]>> = HashMap::new();
+ let mut tmp_paths = Vec::new();
+ for (tmp_path, chunk_map) in results {
tmp_paths.extend(tmp_path);
+ for (source_datastore, chunks) in chunk_map {
+ datastore_chunk_map
+ .entry(source_datastore)
+ .or_default()
+ .extend(chunks);
+ }
}
// sorted media_uuid => (sorted file_num => (set of digests)))
@@ -816,17 +827,15 @@ fn restore_list_worker(
info!("All chunks are already present, skip phase 2...");
}
- for (media_uuid, file_chunk_map) in media_file_chunk_map.iter_mut() {
- let media_id = inventory.lookup_media(media_uuid).unwrap();
- let (mut drive, _info) = request_and_load_media(
- &worker,
- &drive_config,
- drive_name,
- &media_id.label,
- notification_mode,
- )?;
- restore_file_chunk_map(worker.clone(), &mut drive, &store_map, file_chunk_map)?;
- }
+ drive_pool.run(media_file_chunk_map, |media, mut file_chunk_map| {
+ restore_file_chunk_map(
+ worker.clone(),
+ &mut media.drive,
+ &store_map,
+ &mut file_chunk_map,
+ &media.log_prefix,
+ )
+ })?;
info!("Phase 3: copy snapshots from temp dir to datastores");
let mut errors = false;
@@ -997,14 +1006,16 @@ fn snapshot_tmpdir(
Ok(path)
}
+#[allow(clippy::too_many_arguments)]
fn restore_snapshots_to_tmpdir(
worker: Arc<WorkerTask>,
store_map: &DataStoreMap,
file_list: &[(u64, String)],
- mut drive: Box<dyn TapeDriver>,
+ drive: &mut Box<dyn TapeDriver>,
media_id: &MediaId,
media_set_uuid: &Uuid,
chunks_list: &mut HashMap<String, HashSet<[u8; 32]>>,
+ log_prefix: &str,
) -> Result<Vec<PathBuf>, Error> {
let mut tmp_paths = Vec::new();
match media_id.media_set_label {
@@ -1030,10 +1041,10 @@ fn restore_snapshots_to_tmpdir(
for (file_num, snapshot_name) in file_list {
let current_file_number = drive.current_file_number()?;
if current_file_number != *file_num {
- info!("was at file {current_file_number}, moving to {file_num}");
+ info!("{log_prefix}was at file {current_file_number}, moving to {file_num}");
drive.move_to_file(*file_num)?;
let current_file_number = drive.current_file_number()?;
- info!("now at file {current_file_number}");
+ info!("{log_prefix}now at file {current_file_number}");
}
let mut reader = drive.read_next_file()?;
@@ -1062,7 +1073,9 @@ fn restore_snapshots_to_tmpdir(
bail!("unexpected snapshot name (expected {snapshot_name}, got {snapshot})");
}
- info!("File {file_num}: snapshot archive {source_datastore}:{snapshot}",);
+ info!(
+ "{log_prefix}File {file_num}: snapshot archive {source_datastore}:{snapshot}",
+ );
let mut decoder =
pxar::decoder::sync::Decoder::from_std(pxar::PxarVariant::Unified(reader))?;
@@ -1070,7 +1083,10 @@ fn restore_snapshots_to_tmpdir(
let target_datastore = match store_map.target_store(&source_datastore) {
Some(datastore) => datastore,
None => {
- warn!("could not find target datastore for {source_datastore}:{snapshot}",);
+ warn!(
+ "{log_prefix}could not find target datastore for \
+ {source_datastore}:{snapshot}",
+ );
continue;
}
};
@@ -1119,14 +1135,15 @@ fn restore_file_chunk_map(
drive: &mut Box<dyn TapeDriver>,
store_map: &DataStoreMap,
file_chunk_map: &mut BTreeMap<u64, HashSet<[u8; 32]>>,
+ log_prefix: &str,
) -> Result<(), Error> {
for (nr, chunk_map) in file_chunk_map.iter_mut() {
let current_file_number = drive.current_file_number()?;
if current_file_number != *nr {
- info!("was at file {current_file_number}, moving to {nr}");
+ info!("{log_prefix}was at file {current_file_number}, moving to {nr}");
drive.move_to_file(*nr)?;
let current_file_number = drive.current_file_number()?;
- info!("now at file {current_file_number}");
+ info!("{log_prefix}now at file {current_file_number}");
}
let mut reader = drive.read_next_file()?;
let header: MediaContentHeader = unsafe { reader.read_le_value()? };
@@ -1143,7 +1160,7 @@ fn restore_file_chunk_map(
let source_datastore = archive_header.store;
- info!("File {nr}: chunk archive for datastore '{source_datastore}'",);
+ info!("{log_prefix}File {nr}: chunk archive for datastore '{source_datastore}'",);
let datastore = store_map.target_store(&source_datastore).ok_or_else(|| {
format_err!("unexpected chunk archive for store: {source_datastore}")
@@ -1154,8 +1171,9 @@ fn restore_file_chunk_map(
reader,
datastore.clone(),
chunk_map,
+ log_prefix,
)?;
- info!("restored {count} chunks");
+ info!("{log_prefix}restored {count} chunks");
}
_ => bail!("unexpected content magic {:?}", header.content_magic),
}
@@ -1169,6 +1187,7 @@ fn restore_partial_chunk_archive<'a>(
reader: Box<dyn 'a + TapeRead>,
datastore: Arc<DataStore>,
chunk_list: &mut HashSet<[u8; 32]>,
+ log_prefix: &str,
) -> Result<usize, Error> {
let mut decoder = ChunkArchiveDecoder::new(reader);
@@ -1217,7 +1236,7 @@ fn restore_partial_chunk_archive<'a>(
let elapsed = start_time.elapsed()?.as_secs_f64();
let bytes = bytes.load(std::sync::atomic::Ordering::SeqCst) as f64;
info!(
- "restored {} ({:.2}/s)",
+ "{log_prefix}restored {} ({:.2}/s)",
HumanByte::new_decimal(bytes),
HumanByte::new_decimal(bytes / elapsed),
);
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH proxmox-backup 3/4] close #7333: tape: restore: allow using several drives in parallel
2026-11-12 11:37 [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 1/4] tape: drive: add a pool to use several drives for one task Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 2/4] tape: restore: use a drive pool for the selective restore Jakob-Tobias Winter
@ 2026-11-12 11:37 ` Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 4/4] ui: tape restore: allow selecting several drives Jakob-Tobias Winter
3 siblings, 0 replies; 5+ messages in thread
From: Jakob-Tobias Winter @ 2026-11-12 11:37 UTC (permalink / raw)
To: pbs-devel; +Cc: Jakob-Tobias Winter
Restoring a media set that spans several tapes from a single drive
takes long, even if other drives of the library are idle. Add the
'drives' parameter to the restore API and the proxmox-tape CLI, a list
of drives that replaces 'drive'. The old parameter still works, but is
deprecated. Several drives have to be in the same changer, as they
share the media of the restore. All drives are checked for the
Tape.Read privilege, locked and marked as in use for the whole task.
A selective restore then reads the media of its first two phases in
parallel. A full restore cannot simply do the same: a tape backup
writes the chunks of a snapshot before its snapshot archive, possibly
to an earlier media, so with several drives a snapshot could be added
before all of its chunks are restored, and an aborted restore would
leave it behind with chunks missing. So with several drives, a full
restore uses the phases of the selective restore for all snapshots,
which only add the snapshots once all chunks are restored. This needs
the media set catalog, the snapshots only show up at the end of the
restore, and each media is loaded twice, once for the small snapshot
archives and once for the chunks. With a single drive, a full restore
works as before.
Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
---
changes since the RFC:
- 'drives' replaces 'drive', which stays as a deprecated parameter,
instead of adding 'additional-drives'
- no limit for the number of drives
- several drives have to be in the same changer
- a set instead of a search in the list to find a drive given twice
- the list of snapshots is not optional for the worker anymore, an
empty one tells that everything is restored
src/api2/tape/restore.rs | 85 +++++++++++++++++++++++++++++++++-------
src/bin/proxmox-tape.rs | 15 ++++++-
2 files changed, 85 insertions(+), 15 deletions(-)
diff --git a/src/api2/tape/restore.rs b/src/api2/tape/restore.rs
index 9f0111216..588ac1a5c 100644
--- a/src/api2/tape/restore.rs
+++ b/src/api2/tape/restore.rs
@@ -41,7 +41,8 @@ use crate::backup::check_ns_modification_privs;
use crate::tape::{
Inventory, MediaCatalog, MediaId, MediaSet, MediaSetCatalog, TAPE_STATUS_DIR,
drive::{
- DrivePool, TapeDriver, lock_tape_device, request_and_load_media, set_tape_device_state,
+ DrivePool, TapeDriver, lock_tape_device, media_changer, request_and_load_media,
+ set_tape_device_state,
},
file_formats::{
CatalogArchiveHeader, ChunkArchiveDecoder, ChunkArchiveHeader,
@@ -285,6 +286,18 @@ pub const ROUTER: Router = Router::new().post(&API_METHOD_RESTORE);
},
drive: {
schema: DRIVE_NAME_SCHEMA,
+ optional: true,
+ },
+ drives: {
+ description: "List of drives to use. Several drives have to be in the same \
+ changer, their media are then read in parallel. Replaces the deprecated \
+ 'drive'.",
+ type: Array,
+ optional: true,
+ min_length: 1,
+ items: {
+ schema: DRIVE_NAME_SCHEMA,
+ },
},
"media-set": {
description: "Media set UUID.",
@@ -318,8 +331,9 @@ pub const ROUTER: Router = Router::new().post(&API_METHOD_RESTORE);
access: {
// Note: parameters are no uri parameter, so we need to test inside function body
description: "The user needs Tape.Read privilege on /tape/pool/{pool} and \
- /tape/device/{drive}, Datastore.Backup privilege on /datastore/{store}/[{namespace}], \
- Datastore.Modify privileges to create namespaces (if they don't exist).",
+ /tape/device/{drive} for each drive, Datastore.Backup privilege on \
+ /datastore/{store}/[{namespace}], Datastore.Modify privileges to create namespaces \
+ (if they don't exist).",
permission: &Permission::Anybody,
},
)]
@@ -327,7 +341,8 @@ pub const ROUTER: Router = Router::new().post(&API_METHOD_RESTORE);
#[allow(clippy::too_many_arguments)]
pub fn restore(
store: String,
- drive: String,
+ drive: Option<String>,
+ drives: Option<Vec<String>>,
namespaces: Option<Vec<String>>,
media_set: String,
notify_user: Option<Userid>,
@@ -374,7 +389,20 @@ pub fn restore(
}
}
}
- user_info.check_privs(&auth_id, &["tape", "device", &drive], PRIV_TAPE_READ, false)?;
+ // TODO: remove the deprecated 'drive' with PBS 5
+ let drives = match (drive, drives) {
+ (Some(drive), None) => vec![drive],
+ (None, Some(drives)) => drives,
+ (Some(_), Some(_)) => bail!("'drive' and 'drives' are mutually exclusive"),
+ (None, None) => bail!("no drives given"),
+ };
+ let mut unique = HashSet::new();
+ for drive in drives.iter() {
+ if !unique.insert(drive) {
+ bail!("drive '{drive}' given more than once");
+ }
+ user_info.check_privs(&auth_id, &["tape", "device", drive], PRIV_TAPE_READ, false)?;
+ }
let media_set_uuid = media_set.parse()?;
@@ -387,8 +415,25 @@ pub fn restore(
let (drive_config, _digest) = pbs_config::drive::config()?;
+ if drives.len() > 1 {
+ // the drives share the media, so all of them have to reach the same slots
+ let mut changers = HashSet::new();
+ for drive in drives.iter() {
+ match media_changer(&drive_config, drive)? {
+ Some((_, changer)) => changers.insert(changer),
+ None => bail!("several drives have to be in the same changer, '{drive}' has none"),
+ };
+ }
+ if changers.len() > 1 {
+ bail!("several drives have to be in the same changer");
+ }
+ }
+
// early check/lock before starting worker
- let drive_lock = lock_tape_device(&drive_config, &drive)?;
+ let drive_locks = drives
+ .iter()
+ .map(|drive| lock_tape_device(&drive_config, drive))
+ .collect::<Result<Vec<_>, _>>()?;
let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
@@ -404,23 +449,28 @@ pub fn restore(
auth_id.to_string(),
to_stdout,
move |worker| {
- let _drive_lock = drive_lock; // keep lock guard
+ let _drive_locks = drive_locks; // keep lock guards
- set_tape_device_state(&drive, &worker.upid().to_string())?;
+ for drive in drives.iter() {
+ set_tape_device_state(drive, &worker.upid().to_string())?;
+ }
let restore_owner = owner.as_ref().unwrap_or(&auth_id);
info!("Mediaset '{media_set}'");
info!("Pool: {pool}");
- let res = if snapshots.is_some() || namespaces {
+ // the chunks of a snapshot can be on an earlier media that another drive did not
+ // restore yet, so with several drives a full restore uses the phases of the selective
+ // restore, which only add the snapshots after all chunks are restored
+ let res = if snapshots.is_some() || namespaces || drives.len() > 1 {
restore_list_worker(
worker,
snapshots.unwrap_or_default(),
inventory,
media_set_uuid,
drive_config,
- std::slice::from_ref(&drive),
+ &drives,
store_map,
restore_owner,
¬ification_mode,
@@ -433,7 +483,7 @@ pub fn restore(
inventory,
media_set_uuid,
drive_config,
- &drive,
+ &drives[0],
store_map,
restore_owner,
¬ification_mode,
@@ -443,8 +493,10 @@ pub fn restore(
if res.is_ok() {
info!("Restore mediaset '{media_set}' done");
}
- if let Err(err) = set_tape_device_state(&drive, "") {
- info!("could not unset drive state for {drive}: {err}");
+ for drive in drives.iter() {
+ if let Err(err) = set_tape_device_state(drive, "") {
+ info!("could not unset drive state for {drive}: {err}");
+ }
}
res
@@ -652,11 +704,16 @@ fn restore_list_worker(
let res = proxmox_lang::try_block!({
// phase 0
+ // without snapshots and namespace mapping, this is a full restore using several drives
+ let restore_all = drives.len() > 1 && snapshots.is_empty() && store_map.ns_map.is_none();
let snapshots = if snapshots.is_empty() {
let mut restorable = Vec::new();
// restore source namespaces
for (store, snapshot) in catalog.list_snapshots() {
let (ns, dir) = match parse_ns_and_snapshot(snapshot) {
+ Ok((ns, dir)) if restore_all && store_map.target_store(store).is_some() => {
+ (ns, dir)
+ }
Ok((ns, dir)) if store_map.has_full_mapping(store, &ns) => (ns, dir),
Err(err) => {
warn!("couldn't parse snapshot {snapshot} - {err}");
@@ -671,7 +728,7 @@ fn restore_list_worker(
&snapshot,
&ns,
&dir,
- false,
+ restore_all,
&user_info,
auth_id,
restore_owner,
diff --git a/src/bin/proxmox-tape.rs b/src/bin/proxmox-tape.rs
index dcbf2d0fb..504e6e380 100644
--- a/src/bin/proxmox-tape.rs
+++ b/src/bin/proxmox-tape.rs
@@ -906,6 +906,17 @@ async fn backup(mut param: Value) -> Result<(), Error> {
schema: DRIVE_NAME_SCHEMA,
optional: true,
},
+ drives: {
+ description: "List of drives to use. Several drives have to be in the same \
+ changer, their media are then read in parallel. Replaces the deprecated \
+ 'drive'.",
+ type: Array,
+ optional: true,
+ min_length: 1,
+ items: {
+ schema: DRIVE_NAME_SCHEMA,
+ },
+ },
"media-set": {
description: "Media set UUID.",
type: String,
@@ -939,7 +950,9 @@ async fn restore(mut param: Value) -> Result<(), Error> {
let (config, _digest) = pbs_config::drive::config()?;
- param["drive"] = extract_drive_name(&mut param, &config)?.into();
+ if param["drives"].is_null() {
+ param["drive"] = extract_drive_name(&mut param, &config)?.into();
+ }
let client = connect_to_localhost()?;
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH proxmox-backup 4/4] ui: tape restore: allow selecting several drives
2026-11-12 11:37 [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring Jakob-Tobias Winter
` (2 preceding siblings ...)
2026-11-12 11:37 ` [PATCH proxmox-backup 3/4] close #7333: tape: restore: allow using several drives in parallel Jakob-Tobias Winter
@ 2026-11-12 11:37 ` Jakob-Tobias Winter
3 siblings, 0 replies; 5+ messages in thread
From: Jakob-Tobias Winter @ 2026-11-12 11:37 UTC (permalink / raw)
To: pbs-devel; +Cc: Jakob-Tobias Winter
Turn the drive selector of the restore window into a multi-select
field and submit it as the new 'drives' parameter, so that a restore
can read several media in parallel. Like the API, the selector does
not accept several drives that are not in the same changer.
Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
---
changes since the RFC:
- one multi-select field that is sent as 'drives', instead of a
second field for additional drives
- the drive selector does not accept several drives that are not in
the same changer
www/tape/form/DriveSelector.js | 17 +++++++++++++++++
www/tape/window/TapeRestore.js | 5 +++--
2 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/www/tape/form/DriveSelector.js b/www/tape/form/DriveSelector.js
index ca4188c25..0a5515493 100644
--- a/www/tape/form/DriveSelector.js
+++ b/www/tape/form/DriveSelector.js
@@ -52,6 +52,23 @@ Ext.define('PBS.form.DriveSelector', {
],
},
+ // several drives can only be used together if they are in the same changer
+ validator: function (value) {
+ let me = this;
+
+ let drives = me.multiSelect ? me.getValue() || [] : [];
+ if (drives.length > 1) {
+ let changers = drives.map(
+ (drive) => me.store.getAt(me.store.findExact('name', drive))?.data.changer,
+ );
+ if (changers.some((changer) => !changer || changer !== changers[0])) {
+ return gettext('Several drives have to be in the same changer');
+ }
+ }
+
+ return me.callParent([value]);
+ },
+
initComponent: function () {
let me = this;
diff --git a/www/tape/window/TapeRestore.js b/www/tape/window/TapeRestore.js
index 8ee4f17a9..47efab2de 100644
--- a/www/tape/window/TapeRestore.js
+++ b/www/tape/window/TapeRestore.js
@@ -475,9 +475,10 @@ Ext.define('PBS.TapeManagement.TapeRestoreWindow', {
column2: [
{
xtype: 'pbsDriveSelector',
- name: 'drive',
- fieldLabel: gettext('Drive'),
+ name: 'drives',
+ fieldLabel: gettext('Drives'),
labelWidth: 120,
+ multiSelect: true,
},
{
xtype: 'pbsDataStoreSelector',
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-12 11:41 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-12 11:37 [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 1/4] tape: drive: add a pool to use several drives for one task Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 2/4] tape: restore: use a drive pool for the selective restore Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 3/4] close #7333: tape: restore: allow using several drives in parallel Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 4/4] ui: tape restore: allow selecting several drives Jakob-Tobias Winter
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.