From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 219B21FF0B6 for ; Thu, 12 Nov 2026 12:41:18 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 283C12167B; Thu, 12 Nov 2026 12:41:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stuttgart.de; s=s1022; t=1794483670; bh=d9eB3YKOyBcP3dLoMQ2eeIEEW3h6vRIKY+AkQOpRGhI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iN5JsoPXAEvp6M4T5JKspecKqRNc809ddakcjCkXt/MdFZ8EJM6e5rsHcYDWeg44N fE0x/Y34EQGNpLUv0yIaEZToQo2FSJ/jqJZnBm062hKN/mo9+35+Y9mNenFmO1Md6L vZfCjVIj4sI+3MMH8unW9BH7OwZkT2Rh1AtEA7SxmOfSir42BN0MFvUq4GSeb2vs6a ERMxXck6mRcKF1tfAtznaganKfZBsCERBgTYwRpe5cRTTWMksugx0E3G37dW4KENEy jhmyb+Woe5EsH7t2UL4+wWAJtmXKcrXB7EdDpsm/mLzUxjnaFSU44x/Dtq4+5BrXFO 2WM8ur2R+5zow== From: Jakob-Tobias Winter To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup 1/4] tape: drive: add a pool to use several drives for one task Date: Thu, 12 Nov 2026 12:37:34 +0100 Message-ID: <20261112113737.748541-2-tobias@stuttgart.de> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20261112113737.748541-1-tobias@stuttgart.de> References: <20261112113737.748541-1-tobias@stuttgart.de> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.035 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: TEMQNUCOV7RQFX3MVY4P2ZDBNYZPNUWW X-Message-ID-Hash: TEMQNUCOV7RQFX3MVY4P2ZDBNYZPNUWW X-MailFrom: prvs=f560d3371a=tobias@stuttgart.de X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Jakob-Tobias Winter X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Restoring a media set loads one tape after the other into a single drive, even if the library has more drives that are idle. In preparation for using them in parallel, add a pool of drives that runs a list of jobs, each needing one media, on all of its drives. With a single drive, the jobs run in the given order in the calling thread, exactly like the existing loops that load one media after the other. With several drives, the media in the drives of the pool are unloaded first. If a needed media is loaded in another drive, it cannot be loaded, so the task fails right away. Then each drive runs in its own thread, where it takes the next media in order, loads it, runs the job and unloads it again, also if the job failed or was aborted. Changer operations are serialized, so two drives do not pick the same free slot when unloading a media. A drive rewinds and ejects its media before that, so the others only wait while the changer moves a media. The first error aborts the task, which stops the other drives, and is returned once all of them finished. As the lines of all drives are interleaved in the task log, their messages start with the name of the drive, and the jobs get this prefix for their own messages. With a single drive, the prefix is empty, so the log stays as it is. Suggested-by: Dominik Csapak Signed-off-by: Jakob-Tobias Winter --- changes since the RFC: - each drive takes the next media in order instead of preferring the one that is loaded in it: all drives are unloaded first, and the task fails if a needed media is loaded in a drive that is not used - a drive unloads its media after the job, also if the job failed or was aborted, and ejects it before it waits for the changer - a failing drive aborts the task instead of setting a stop flag of its own, so the separate task context and the panic guard are gone - panic on a poisoned lock instead of ignoring it - the jobs are a map with the media uuids as keys, and the pool looks up the labels in the inventory - the pool keeps the drive and hands it to the job by reference, to eject the media afterwards src/tape/drive/mod.rs | 40 ++++++- src/tape/drive/pool.rs | 259 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 293 insertions(+), 6 deletions(-) create mode 100644 src/tape/drive/pool.rs diff --git a/src/tape/drive/mod.rs b/src/tape/drive/mod.rs index fa84fd54a..45f75b768 100644 --- a/src/tape/drive/mod.rs +++ b/src/tape/drive/mod.rs @@ -1,6 +1,7 @@ //! Tape drivers use std::path::PathBuf; +use std::sync::Mutex; use anyhow::{Error, bail, format_err}; use nix::fcntl::OFlag; @@ -39,6 +40,9 @@ mod virtual_tape; mod lto; pub use lto::*; +mod pool; +pub use pool::*; + /// Tape driver interface pub trait TapeDriver { /// Flush all data to the tape @@ -378,11 +382,28 @@ pub fn request_and_load_media( drive: &str, label: &MediaLabel, notification_mode: &TapeNotificationMode, +) -> Result<(Box, MediaId), Error> { + request_and_load_media_locked(worker, config, drive, label, notification_mode, None, "") +} + +/// Like [`request_and_load_media`], but holds `changer_lock` while the changer moves media and +/// starts each log message with `log_prefix`. +/// +/// This serializes the changer operations of several drives used by the same task, and tells +/// their messages apart in the task log. +pub fn request_and_load_media_locked( + worker: &dyn WorkerTaskContext, + config: &SectionConfigData, + drive: &str, + label: &MediaLabel, + notification_mode: &TapeNotificationMode, + changer_lock: Option<&Mutex<()>>, + log_prefix: &str, ) -> Result<(Box, MediaId), Error> { let check_label = |handle: &mut dyn TapeDriver, uuid: &proxmox_uuid::Uuid| { if let Ok((Some(media_id), _)) = handle.read_label() { info!( - "found media label {} ({})", + "{log_prefix}found media label {} ({})", media_id.label.label_text, media_id.label.uuid, ); @@ -422,7 +443,7 @@ pub fn request_and_load_media( new: TapeRequestError| -> Result<(), Error> { if new != *old { - info!("{new}"); + info!("{log_prefix}{new}"); let (device_type, device) = if let Some(changer) = changer { ("changer", changer.as_str()) } else { @@ -430,7 +451,8 @@ pub fn request_and_load_media( }; info!( - "Please insert media '{label_text}' into {device_type} '{device}'" + "{log_prefix}Please insert media '{label_text}' into {device_type} \ + '{device}'" ); send_load_media_notification( notification_mode, @@ -455,13 +477,19 @@ pub fn request_and_load_media( std::thread::sleep(std::time::Duration::from_millis(100)); } } else if drive_config.changer.is_none() { - info!("Checking for media '{label_text}' in drive '{drive}'"); + info!( + "{log_prefix}Checking for media '{label_text}' in drive '{drive}'" + ); } else { - info!("trying to load media '{label_text}' into drive '{drive}'"); + info!( + "{log_prefix}trying to load media '{label_text}' into drive \ + '{drive}'" + ); } if drive_config.changer.is_some() { let mut changer = MtxMediaChanger::with_drive_config(&drive_config)?; + let _changer_guard = changer_lock.map(|lock| lock.lock().unwrap()); if let Err(err) = changer.load_media(&label_text) { update_and_log_request_error( &mut last_error, @@ -485,7 +513,7 @@ pub fn request_and_load_media( let request_error = match handle.read_label() { Ok((Some(media_id), _)) if media_id.label.uuid == label.uuid => { info!( - "found media label {} ({})", + "{log_prefix}found media label {} ({})", media_id.label.label_text, media_id.label.uuid.to_string(), ); diff --git a/src/tape/drive/pool.rs b/src/tape/drive/pool.rs new file mode 100644 index 000000000..3906769e5 --- /dev/null +++ b/src/tape/drive/pool.rs @@ -0,0 +1,259 @@ +//! Use several tape drives for one task + +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::sync::{Arc, Mutex}; + +use anyhow::{Error, bail}; +use tracing::{info, warn}; + +use proxmox_log::LogContext; +use proxmox_rest_server::WorkerTask; +use proxmox_section_config::SectionConfigData; +use proxmox_uuid::Uuid; +use proxmox_worker_task::WorkerTaskContext; + +use pbs_tape::ElementStatus; + +use crate::tape::{Inventory, MediaId, TapeNotificationMode, file_formats::MediaLabel}; + +use super::{TapeDriver, media_changer, request_and_load_media, request_and_load_media_locked}; + +/// A media loaded into a drive of a [`DrivePool`]. +pub struct LoadedMedia { + pub drive: Box, + pub media_id: MediaId, + /// The start of the job's log messages, names the drive if the pool has several. + pub log_prefix: String, +} + +/// A set of drives that work on the media of one task in parallel. +pub struct DrivePool<'a> { + worker: Arc, + config: &'a SectionConfigData, + inventory: &'a Inventory, + drives: &'a [String], + notification_mode: &'a TapeNotificationMode, + /// Serializes the changer operations of all drives. + changer_lock: Mutex<()>, +} + +impl<'a> DrivePool<'a> { + /// Create a pool of `drives`, which the caller has to lock. + pub fn new( + worker: Arc, + config: &'a SectionConfigData, + inventory: &'a Inventory, + drives: &'a [String], + notification_mode: &'a TapeNotificationMode, + ) -> Self { + Self { + worker, + config, + inventory, + drives, + notification_mode, + changer_lock: Mutex::new(()), + } + } + + /// Load each media of `jobs` into a drive and run `func` with its job. + /// + /// With a single drive, the jobs run in the order of the map in the calling thread. + /// + /// With several drives, the media in the drives are unloaded first, and a media of a job that + /// is loaded in another drive is an error. Then each drive runs in its own thread, where it + /// takes the first job that is left, loads its media, runs `func` and unloads the media + /// again, also if `func` failed. The first error aborts the task, which stops the jobs on + /// all drives, and is returned once all of them finished. + /// + /// The results are returned in the order the jobs finished. + pub fn run(&self, jobs: BTreeMap, func: F) -> Result, Error> + where + T: Send, + R: Send, + F: Fn(&mut LoadedMedia, T) -> Result + Sync, + { + if let [drive_name] = self.drives { + let mut results = Vec::with_capacity(jobs.len()); + for (media_uuid, job) in jobs { + let (drive, media_id) = request_and_load_media( + &*self.worker, + self.config, + drive_name, + self.label(&media_uuid)?, + self.notification_mode, + )?; + let mut media = LoadedMedia { + drive, + media_id, + log_prefix: String::new(), + }; + results.push(func(&mut media, job)?); + } + return Ok(results); + } + + self.unload_drives(&jobs)?; + + let queue = Mutex::new(jobs); + let results = Mutex::new(Vec::new()); + let first_error = Mutex::new(None); + let log_context = LogContext::current(); + + std::thread::scope(|scope| { + for drive_name in self.drives { + let (queue, results, first_error, func) = (&queue, &results, &first_error, &func); + let log_context = log_context.clone(); + + scope.spawn(move || { + let run = || { + let mut loaded = None; + let res = self.drive_loop(drive_name, queue, results, func, &mut loaded); + if let Err(err) = res { + let mut first_error = first_error.lock().unwrap(); + if first_error.is_none() { + if !self.worker.abort_requested() { + warn!( + "drive '{drive_name}': failed, stopping all drives - {err}" + ); + self.worker.request_abort(); + } + *first_error = Some(err); + } + } + // a job that failed or was aborted left its media in the drive + let unloaded = loaded.map(|media| self.unload_media(drive_name, media)); + if let Some(Err(err)) = unloaded { + warn!("drive '{drive_name}': unable to unload media - {err}"); + } + }; + match log_context { + Some(log_context) => log_context.sync_scope(run), + None => run(), + } + }); + } + }); + + if let Some(err) = first_error.into_inner().unwrap() { + return Err(err); + } + + Ok(results.into_inner().unwrap()) + } + + fn label(&self, media_uuid: &Uuid) -> Result<&'a MediaLabel, Error> { + match self.inventory.lookup_media(media_uuid) { + Some(media_id) => Ok(&media_id.label), + None => bail!("unable to lookup media {media_uuid}"), + } + } + + /// Unload the media of all drives, then make sure that no media of `jobs` is loaded in a + /// drive that is not part of this pool, as it cannot be loaded from there. + /// + /// Drives without a changer are skipped. + fn unload_drives(&self, jobs: &BTreeMap) -> Result<(), Error> { + let mut changers = HashMap::new(); + + for drive_name in self.drives { + let Some((mut changer, changer_name)) = media_changer(self.config, drive_name)? else { + continue; + }; + let mut status = changer.status()?; + let loaded = status + .drives + .get(changer.drive_number() as usize) + .is_some_and(|drive| !matches!(drive.status, ElementStatus::Empty)); + if loaded { + info!("drive '{drive_name}': unloading media"); + status = changer.unload_media(None)?; + } + changers.insert(changer_name, status); + } + + let mut needed = HashSet::new(); + for media_uuid in jobs.keys() { + needed.insert(self.label(media_uuid)?.label_text.as_str()); + } + + for (changer_name, status) in changers { + for (drive_number, drive) in status.drives.iter().enumerate() { + let ElementStatus::VolumeTag(ref tag) = drive.status else { + continue; + }; + if needed.contains(tag.as_str()) { + bail!( + "media '{tag}' is loaded in drive {drive_number} of changer \ + '{changer_name}', which this task does not use" + ); + } + } + } + + Ok(()) + } + + fn drive_loop( + &self, + drive_name: &str, + queue: &Mutex>, + results: &Mutex>, + func: &F, + loaded: &mut Option, + ) -> Result<(), Error> + where + F: Fn(&mut LoadedMedia, T) -> Result, + { + // the lines of all drives are interleaved in the task log + let log_prefix = format!("drive '{drive_name}': "); + + loop { + self.worker.check_abort()?; + + let Some((media_uuid, job)) = queue.lock().unwrap().pop_first() else { + return Ok(()); + }; + let label = self.label(&media_uuid)?; + + let (drive, media_id) = request_and_load_media_locked( + &*self.worker, + self.config, + drive_name, + label, + self.notification_mode, + Some(&self.changer_lock), + &log_prefix, + )?; + // stays in `loaded` if the job fails, the caller then unloads the media + let media = loaded.insert(LoadedMedia { + drive, + media_id, + log_prefix: log_prefix.clone(), + }); + let result = func(media, job)?; + results.lock().unwrap().push(result); + + if let Some(media) = loaded.take() { + self.unload_media(drive_name, media)?; + } + } + } + + /// Unload the media from a drive after its job, if the drive is in a changer. + fn unload_media(&self, drive_name: &str, mut media: LoadedMedia) -> Result<(), Error> { + let Some((mut changer, _)) = media_changer(self.config, drive_name)? else { + return Ok(()); + }; + info!( + "{}unloading media '{}'", + media.log_prefix, media.media_id.label.label_text + ); + // rewind and eject first, so that the other drives only wait for the changer + media.drive.eject_media()?; + drop(media); + let _changer_guard = self.changer_lock.lock().unwrap(); + changer.unload_media(None)?; + Ok(()) + } +} -- 2.53.0