From: Jakob-Tobias Winter <tobias@stuttgart.de>
To: pbs-devel@lists.proxmox.com
Cc: Jakob-Tobias Winter <tobias@stuttgart.de>
Subject: [PATCH proxmox-backup 1/4] tape: drive: add a pool to use several drives for one task
Date: Thu, 12 Nov 2026 12:37:34 +0100 [thread overview]
Message-ID: <20261112113737.748541-2-tobias@stuttgart.de> (raw)
In-Reply-To: <20261112113737.748541-1-tobias@stuttgart.de>
Restoring a media set loads one tape after the other into a single
drive, even if the library has more drives that are idle. In
preparation for using them in parallel, add a pool of drives that runs
a list of jobs, each needing one media, on all of its drives.
With a single drive, the jobs run in the given order in the calling
thread, exactly like the existing loops that load one media after the
other.
With several drives, the media in the drives of the pool are unloaded
first. If a needed media is loaded in another drive, it cannot be
loaded, so the task fails right away. Then each drive runs in its own
thread, where it takes the next media in order, loads it, runs the job
and unloads it again, also if the job failed or was aborted. Changer
operations are serialized, so two drives do not pick the same free
slot when unloading a media. A drive rewinds and ejects its media
before that, so the others only wait while the changer moves a media.
The first error aborts the task, which stops the other drives, and is
returned once all of them finished.
As the lines of all drives are interleaved in the task log, their
messages start with the name of the drive, and the jobs get this
prefix for their own messages. With a single drive, the prefix is
empty, so the log stays as it is.
Suggested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
---
changes since the RFC:
- each drive takes the next media in order instead of preferring the
one that is loaded in it: all drives are unloaded first, and the
task fails if a needed media is loaded in a drive that is not used
- a drive unloads its media after the job, also if the job failed or
was aborted, and ejects it before it waits for the changer
- a failing drive aborts the task instead of setting a stop flag of
its own, so the separate task context and the panic guard are gone
- panic on a poisoned lock instead of ignoring it
- the jobs are a map with the media uuids as keys, and the pool looks
up the labels in the inventory
- the pool keeps the drive and hands it to the job by reference, to
eject the media afterwards
src/tape/drive/mod.rs | 40 ++++++-
src/tape/drive/pool.rs | 259 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 293 insertions(+), 6 deletions(-)
create mode 100644 src/tape/drive/pool.rs
diff --git a/src/tape/drive/mod.rs b/src/tape/drive/mod.rs
index fa84fd54a..45f75b768 100644
--- a/src/tape/drive/mod.rs
+++ b/src/tape/drive/mod.rs
@@ -1,6 +1,7 @@
//! Tape drivers
use std::path::PathBuf;
+use std::sync::Mutex;
use anyhow::{Error, bail, format_err};
use nix::fcntl::OFlag;
@@ -39,6 +40,9 @@ mod virtual_tape;
mod lto;
pub use lto::*;
+mod pool;
+pub use pool::*;
+
/// Tape driver interface
pub trait TapeDriver {
/// Flush all data to the tape
@@ -378,11 +382,28 @@ pub fn request_and_load_media(
drive: &str,
label: &MediaLabel,
notification_mode: &TapeNotificationMode,
+) -> Result<(Box<dyn TapeDriver>, MediaId), Error> {
+ request_and_load_media_locked(worker, config, drive, label, notification_mode, None, "")
+}
+
+/// Like [`request_and_load_media`], but holds `changer_lock` while the changer moves media and
+/// starts each log message with `log_prefix`.
+///
+/// This serializes the changer operations of several drives used by the same task, and tells
+/// their messages apart in the task log.
+pub fn request_and_load_media_locked(
+ worker: &dyn WorkerTaskContext,
+ config: &SectionConfigData,
+ drive: &str,
+ label: &MediaLabel,
+ notification_mode: &TapeNotificationMode,
+ changer_lock: Option<&Mutex<()>>,
+ log_prefix: &str,
) -> Result<(Box<dyn TapeDriver>, MediaId), Error> {
let check_label = |handle: &mut dyn TapeDriver, uuid: &proxmox_uuid::Uuid| {
if let Ok((Some(media_id), _)) = handle.read_label() {
info!(
- "found media label {} ({})",
+ "{log_prefix}found media label {} ({})",
media_id.label.label_text, media_id.label.uuid,
);
@@ -422,7 +443,7 @@ pub fn request_and_load_media(
new: TapeRequestError|
-> Result<(), Error> {
if new != *old {
- info!("{new}");
+ info!("{log_prefix}{new}");
let (device_type, device) = if let Some(changer) = changer {
("changer", changer.as_str())
} else {
@@ -430,7 +451,8 @@ pub fn request_and_load_media(
};
info!(
- "Please insert media '{label_text}' into {device_type} '{device}'"
+ "{log_prefix}Please insert media '{label_text}' into {device_type} \
+ '{device}'"
);
send_load_media_notification(
notification_mode,
@@ -455,13 +477,19 @@ pub fn request_and_load_media(
std::thread::sleep(std::time::Duration::from_millis(100));
}
} else if drive_config.changer.is_none() {
- info!("Checking for media '{label_text}' in drive '{drive}'");
+ info!(
+ "{log_prefix}Checking for media '{label_text}' in drive '{drive}'"
+ );
} else {
- info!("trying to load media '{label_text}' into drive '{drive}'");
+ info!(
+ "{log_prefix}trying to load media '{label_text}' into drive \
+ '{drive}'"
+ );
}
if drive_config.changer.is_some() {
let mut changer = MtxMediaChanger::with_drive_config(&drive_config)?;
+ let _changer_guard = changer_lock.map(|lock| lock.lock().unwrap());
if let Err(err) = changer.load_media(&label_text) {
update_and_log_request_error(
&mut last_error,
@@ -485,7 +513,7 @@ pub fn request_and_load_media(
let request_error = match handle.read_label() {
Ok((Some(media_id), _)) if media_id.label.uuid == label.uuid => {
info!(
- "found media label {} ({})",
+ "{log_prefix}found media label {} ({})",
media_id.label.label_text,
media_id.label.uuid.to_string(),
);
diff --git a/src/tape/drive/pool.rs b/src/tape/drive/pool.rs
new file mode 100644
index 000000000..3906769e5
--- /dev/null
+++ b/src/tape/drive/pool.rs
@@ -0,0 +1,259 @@
+//! Use several tape drives for one task
+
+use std::collections::{BTreeMap, HashMap, HashSet};
+use std::sync::{Arc, Mutex};
+
+use anyhow::{Error, bail};
+use tracing::{info, warn};
+
+use proxmox_log::LogContext;
+use proxmox_rest_server::WorkerTask;
+use proxmox_section_config::SectionConfigData;
+use proxmox_uuid::Uuid;
+use proxmox_worker_task::WorkerTaskContext;
+
+use pbs_tape::ElementStatus;
+
+use crate::tape::{Inventory, MediaId, TapeNotificationMode, file_formats::MediaLabel};
+
+use super::{TapeDriver, media_changer, request_and_load_media, request_and_load_media_locked};
+
+/// A media loaded into a drive of a [`DrivePool`].
+pub struct LoadedMedia {
+ pub drive: Box<dyn TapeDriver>,
+ pub media_id: MediaId,
+ /// The start of the job's log messages, names the drive if the pool has several.
+ pub log_prefix: String,
+}
+
+/// A set of drives that work on the media of one task in parallel.
+pub struct DrivePool<'a> {
+ worker: Arc<WorkerTask>,
+ config: &'a SectionConfigData,
+ inventory: &'a Inventory,
+ drives: &'a [String],
+ notification_mode: &'a TapeNotificationMode,
+ /// Serializes the changer operations of all drives.
+ changer_lock: Mutex<()>,
+}
+
+impl<'a> DrivePool<'a> {
+ /// Create a pool of `drives`, which the caller has to lock.
+ pub fn new(
+ worker: Arc<WorkerTask>,
+ config: &'a SectionConfigData,
+ inventory: &'a Inventory,
+ drives: &'a [String],
+ notification_mode: &'a TapeNotificationMode,
+ ) -> Self {
+ Self {
+ worker,
+ config,
+ inventory,
+ drives,
+ notification_mode,
+ changer_lock: Mutex::new(()),
+ }
+ }
+
+ /// Load each media of `jobs` into a drive and run `func` with its job.
+ ///
+ /// With a single drive, the jobs run in the order of the map in the calling thread.
+ ///
+ /// With several drives, the media in the drives are unloaded first, and a media of a job that
+ /// is loaded in another drive is an error. Then each drive runs in its own thread, where it
+ /// takes the first job that is left, loads its media, runs `func` and unloads the media
+ /// again, also if `func` failed. The first error aborts the task, which stops the jobs on
+ /// all drives, and is returned once all of them finished.
+ ///
+ /// The results are returned in the order the jobs finished.
+ pub fn run<T, R, F>(&self, jobs: BTreeMap<Uuid, T>, func: F) -> Result<Vec<R>, Error>
+ where
+ T: Send,
+ R: Send,
+ F: Fn(&mut LoadedMedia, T) -> Result<R, Error> + Sync,
+ {
+ if let [drive_name] = self.drives {
+ let mut results = Vec::with_capacity(jobs.len());
+ for (media_uuid, job) in jobs {
+ let (drive, media_id) = request_and_load_media(
+ &*self.worker,
+ self.config,
+ drive_name,
+ self.label(&media_uuid)?,
+ self.notification_mode,
+ )?;
+ let mut media = LoadedMedia {
+ drive,
+ media_id,
+ log_prefix: String::new(),
+ };
+ results.push(func(&mut media, job)?);
+ }
+ return Ok(results);
+ }
+
+ self.unload_drives(&jobs)?;
+
+ let queue = Mutex::new(jobs);
+ let results = Mutex::new(Vec::new());
+ let first_error = Mutex::new(None);
+ let log_context = LogContext::current();
+
+ std::thread::scope(|scope| {
+ for drive_name in self.drives {
+ let (queue, results, first_error, func) = (&queue, &results, &first_error, &func);
+ let log_context = log_context.clone();
+
+ scope.spawn(move || {
+ let run = || {
+ let mut loaded = None;
+ let res = self.drive_loop(drive_name, queue, results, func, &mut loaded);
+ if let Err(err) = res {
+ let mut first_error = first_error.lock().unwrap();
+ if first_error.is_none() {
+ if !self.worker.abort_requested() {
+ warn!(
+ "drive '{drive_name}': failed, stopping all drives - {err}"
+ );
+ self.worker.request_abort();
+ }
+ *first_error = Some(err);
+ }
+ }
+ // a job that failed or was aborted left its media in the drive
+ let unloaded = loaded.map(|media| self.unload_media(drive_name, media));
+ if let Some(Err(err)) = unloaded {
+ warn!("drive '{drive_name}': unable to unload media - {err}");
+ }
+ };
+ match log_context {
+ Some(log_context) => log_context.sync_scope(run),
+ None => run(),
+ }
+ });
+ }
+ });
+
+ if let Some(err) = first_error.into_inner().unwrap() {
+ return Err(err);
+ }
+
+ Ok(results.into_inner().unwrap())
+ }
+
+ fn label(&self, media_uuid: &Uuid) -> Result<&'a MediaLabel, Error> {
+ match self.inventory.lookup_media(media_uuid) {
+ Some(media_id) => Ok(&media_id.label),
+ None => bail!("unable to lookup media {media_uuid}"),
+ }
+ }
+
+ /// Unload the media of all drives, then make sure that no media of `jobs` is loaded in a
+ /// drive that is not part of this pool, as it cannot be loaded from there.
+ ///
+ /// Drives without a changer are skipped.
+ fn unload_drives<T>(&self, jobs: &BTreeMap<Uuid, T>) -> Result<(), Error> {
+ let mut changers = HashMap::new();
+
+ for drive_name in self.drives {
+ let Some((mut changer, changer_name)) = media_changer(self.config, drive_name)? else {
+ continue;
+ };
+ let mut status = changer.status()?;
+ let loaded = status
+ .drives
+ .get(changer.drive_number() as usize)
+ .is_some_and(|drive| !matches!(drive.status, ElementStatus::Empty));
+ if loaded {
+ info!("drive '{drive_name}': unloading media");
+ status = changer.unload_media(None)?;
+ }
+ changers.insert(changer_name, status);
+ }
+
+ let mut needed = HashSet::new();
+ for media_uuid in jobs.keys() {
+ needed.insert(self.label(media_uuid)?.label_text.as_str());
+ }
+
+ for (changer_name, status) in changers {
+ for (drive_number, drive) in status.drives.iter().enumerate() {
+ let ElementStatus::VolumeTag(ref tag) = drive.status else {
+ continue;
+ };
+ if needed.contains(tag.as_str()) {
+ bail!(
+ "media '{tag}' is loaded in drive {drive_number} of changer \
+ '{changer_name}', which this task does not use"
+ );
+ }
+ }
+ }
+
+ Ok(())
+ }
+
+ fn drive_loop<T, R, F>(
+ &self,
+ drive_name: &str,
+ queue: &Mutex<BTreeMap<Uuid, T>>,
+ results: &Mutex<Vec<R>>,
+ func: &F,
+ loaded: &mut Option<LoadedMedia>,
+ ) -> Result<(), Error>
+ where
+ F: Fn(&mut LoadedMedia, T) -> Result<R, Error>,
+ {
+ // the lines of all drives are interleaved in the task log
+ let log_prefix = format!("drive '{drive_name}': ");
+
+ loop {
+ self.worker.check_abort()?;
+
+ let Some((media_uuid, job)) = queue.lock().unwrap().pop_first() else {
+ return Ok(());
+ };
+ let label = self.label(&media_uuid)?;
+
+ let (drive, media_id) = request_and_load_media_locked(
+ &*self.worker,
+ self.config,
+ drive_name,
+ label,
+ self.notification_mode,
+ Some(&self.changer_lock),
+ &log_prefix,
+ )?;
+ // stays in `loaded` if the job fails, the caller then unloads the media
+ let media = loaded.insert(LoadedMedia {
+ drive,
+ media_id,
+ log_prefix: log_prefix.clone(),
+ });
+ let result = func(media, job)?;
+ results.lock().unwrap().push(result);
+
+ if let Some(media) = loaded.take() {
+ self.unload_media(drive_name, media)?;
+ }
+ }
+ }
+
+ /// Unload the media from a drive after its job, if the drive is in a changer.
+ fn unload_media(&self, drive_name: &str, mut media: LoadedMedia) -> Result<(), Error> {
+ let Some((mut changer, _)) = media_changer(self.config, drive_name)? else {
+ return Ok(());
+ };
+ info!(
+ "{}unloading media '{}'",
+ media.log_prefix, media.media_id.label.label_text
+ );
+ // rewind and eject first, so that the other drives only wait for the changer
+ media.drive.eject_media()?;
+ drop(media);
+ let _changer_guard = self.changer_lock.lock().unwrap();
+ changer.unload_media(None)?;
+ Ok(())
+ }
+}
--
2.53.0
next prev parent reply other threads:[~2026-11-12 11:41 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 11:37 [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring Jakob-Tobias Winter
2026-11-12 11:37 ` Jakob-Tobias Winter [this message]
2026-11-12 11:37 ` [PATCH proxmox-backup 2/4] tape: restore: use a drive pool for the selective restore Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 3/4] close #7333: tape: restore: allow using several drives in parallel Jakob-Tobias Winter
2026-11-12 11:37 ` [PATCH proxmox-backup 4/4] ui: tape restore: allow selecting several drives Jakob-Tobias Winter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261112113737.748541-2-tobias@stuttgart.de \
--to=tobias@stuttgart.de \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox