From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id D5E951FF0B6 for ; Thu, 12 Nov 2026 12:41:21 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id AD46D21694; Thu, 12 Nov 2026 12:41:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stuttgart.de; s=s1022; t=1794483671; bh=db2N+TnVLa7Q6S/WnS4M34SKueH16qEueoSlgZirZCc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YLTZh4BBt0P2BtarHBH4pwtsC/JGErGmS8GRFkwhQsBT2LrRDXx9th5+K4ksSYCTs MUaoBqPnB2SPC0PDzE/8gbp9uE8CMJu3rWT3FqspNnJOYkfZF0wSONt9SUXjASmZCc H8QIRajQyi8adv775AP7RvK9ygs349nqcv8wzB0m9vG3bqHGdSPcYLgUZt/9hirELm Z7R6PmV3uDDxrwrTr6aBr4wPrREdvP4hqRTMFSUoUyV5YpvvghNwNC8D1DqXX1+/D+ a5goDy8unsnV0CiuqJ+EH6vwkcfvnX9tCUXtLvwUdRnY6LjWsRn+Id+UzkpGPgS2kY Ns7zwc/P5riRQ== From: Jakob-Tobias Winter To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup 3/4] close #7333: tape: restore: allow using several drives in parallel Date: Thu, 12 Nov 2026 12:37:36 +0100 Message-ID: <20261112113737.748541-4-tobias@stuttgart.de> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20261112113737.748541-1-tobias@stuttgart.de> References: <20261112113737.748541-1-tobias@stuttgart.de> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.120 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: QKWWHD4LDG4SC4TA7KKYDFRNB3TDZGOA X-Message-ID-Hash: QKWWHD4LDG4SC4TA7KKYDFRNB3TDZGOA X-MailFrom: prvs=f560d3371a=tobias@stuttgart.de X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Jakob-Tobias Winter X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Restoring a media set that spans several tapes from a single drive takes long, even if other drives of the library are idle. Add the 'drives' parameter to the restore API and the proxmox-tape CLI, a list of drives that replaces 'drive'. The old parameter still works, but is deprecated. Several drives have to be in the same changer, as they share the media of the restore. All drives are checked for the Tape.Read privilege, locked and marked as in use for the whole task. A selective restore then reads the media of its first two phases in parallel. A full restore cannot simply do the same: a tape backup writes the chunks of a snapshot before its snapshot archive, possibly to an earlier media, so with several drives a snapshot could be added before all of its chunks are restored, and an aborted restore would leave it behind with chunks missing. So with several drives, a full restore uses the phases of the selective restore for all snapshots, which only add the snapshots once all chunks are restored. This needs the media set catalog, the snapshots only show up at the end of the restore, and each media is loaded twice, once for the small snapshot archives and once for the chunks. With a single drive, a full restore works as before. Signed-off-by: Jakob-Tobias Winter --- changes since the RFC: - 'drives' replaces 'drive', which stays as a deprecated parameter, instead of adding 'additional-drives' - no limit for the number of drives - several drives have to be in the same changer - a set instead of a search in the list to find a drive given twice - the list of snapshots is not optional for the worker anymore, an empty one tells that everything is restored src/api2/tape/restore.rs | 85 +++++++++++++++++++++++++++++++++------- src/bin/proxmox-tape.rs | 15 ++++++- 2 files changed, 85 insertions(+), 15 deletions(-) diff --git a/src/api2/tape/restore.rs b/src/api2/tape/restore.rs index 9f0111216..588ac1a5c 100644 --- a/src/api2/tape/restore.rs +++ b/src/api2/tape/restore.rs @@ -41,7 +41,8 @@ use crate::backup::check_ns_modification_privs; use crate::tape::{ Inventory, MediaCatalog, MediaId, MediaSet, MediaSetCatalog, TAPE_STATUS_DIR, drive::{ - DrivePool, TapeDriver, lock_tape_device, request_and_load_media, set_tape_device_state, + DrivePool, TapeDriver, lock_tape_device, media_changer, request_and_load_media, + set_tape_device_state, }, file_formats::{ CatalogArchiveHeader, ChunkArchiveDecoder, ChunkArchiveHeader, @@ -285,6 +286,18 @@ pub const ROUTER: Router = Router::new().post(&API_METHOD_RESTORE); }, drive: { schema: DRIVE_NAME_SCHEMA, + optional: true, + }, + drives: { + description: "List of drives to use. Several drives have to be in the same \ + changer, their media are then read in parallel. Replaces the deprecated \ + 'drive'.", + type: Array, + optional: true, + min_length: 1, + items: { + schema: DRIVE_NAME_SCHEMA, + }, }, "media-set": { description: "Media set UUID.", @@ -318,8 +331,9 @@ pub const ROUTER: Router = Router::new().post(&API_METHOD_RESTORE); access: { // Note: parameters are no uri parameter, so we need to test inside function body description: "The user needs Tape.Read privilege on /tape/pool/{pool} and \ - /tape/device/{drive}, Datastore.Backup privilege on /datastore/{store}/[{namespace}], \ - Datastore.Modify privileges to create namespaces (if they don't exist).", + /tape/device/{drive} for each drive, Datastore.Backup privilege on \ + /datastore/{store}/[{namespace}], Datastore.Modify privileges to create namespaces \ + (if they don't exist).", permission: &Permission::Anybody, }, )] @@ -327,7 +341,8 @@ pub const ROUTER: Router = Router::new().post(&API_METHOD_RESTORE); #[allow(clippy::too_many_arguments)] pub fn restore( store: String, - drive: String, + drive: Option, + drives: Option>, namespaces: Option>, media_set: String, notify_user: Option, @@ -374,7 +389,20 @@ pub fn restore( } } } - user_info.check_privs(&auth_id, &["tape", "device", &drive], PRIV_TAPE_READ, false)?; + // TODO: remove the deprecated 'drive' with PBS 5 + let drives = match (drive, drives) { + (Some(drive), None) => vec![drive], + (None, Some(drives)) => drives, + (Some(_), Some(_)) => bail!("'drive' and 'drives' are mutually exclusive"), + (None, None) => bail!("no drives given"), + }; + let mut unique = HashSet::new(); + for drive in drives.iter() { + if !unique.insert(drive) { + bail!("drive '{drive}' given more than once"); + } + user_info.check_privs(&auth_id, &["tape", "device", drive], PRIV_TAPE_READ, false)?; + } let media_set_uuid = media_set.parse()?; @@ -387,8 +415,25 @@ pub fn restore( let (drive_config, _digest) = pbs_config::drive::config()?; + if drives.len() > 1 { + // the drives share the media, so all of them have to reach the same slots + let mut changers = HashSet::new(); + for drive in drives.iter() { + match media_changer(&drive_config, drive)? { + Some((_, changer)) => changers.insert(changer), + None => bail!("several drives have to be in the same changer, '{drive}' has none"), + }; + } + if changers.len() > 1 { + bail!("several drives have to be in the same changer"); + } + } + // early check/lock before starting worker - let drive_lock = lock_tape_device(&drive_config, &drive)?; + let drive_locks = drives + .iter() + .map(|drive| lock_tape_device(&drive_config, drive)) + .collect::, _>>()?; let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI; @@ -404,23 +449,28 @@ pub fn restore( auth_id.to_string(), to_stdout, move |worker| { - let _drive_lock = drive_lock; // keep lock guard + let _drive_locks = drive_locks; // keep lock guards - set_tape_device_state(&drive, &worker.upid().to_string())?; + for drive in drives.iter() { + set_tape_device_state(drive, &worker.upid().to_string())?; + } let restore_owner = owner.as_ref().unwrap_or(&auth_id); info!("Mediaset '{media_set}'"); info!("Pool: {pool}"); - let res = if snapshots.is_some() || namespaces { + // the chunks of a snapshot can be on an earlier media that another drive did not + // restore yet, so with several drives a full restore uses the phases of the selective + // restore, which only add the snapshots after all chunks are restored + let res = if snapshots.is_some() || namespaces || drives.len() > 1 { restore_list_worker( worker, snapshots.unwrap_or_default(), inventory, media_set_uuid, drive_config, - std::slice::from_ref(&drive), + &drives, store_map, restore_owner, ¬ification_mode, @@ -433,7 +483,7 @@ pub fn restore( inventory, media_set_uuid, drive_config, - &drive, + &drives[0], store_map, restore_owner, ¬ification_mode, @@ -443,8 +493,10 @@ pub fn restore( if res.is_ok() { info!("Restore mediaset '{media_set}' done"); } - if let Err(err) = set_tape_device_state(&drive, "") { - info!("could not unset drive state for {drive}: {err}"); + for drive in drives.iter() { + if let Err(err) = set_tape_device_state(drive, "") { + info!("could not unset drive state for {drive}: {err}"); + } } res @@ -652,11 +704,16 @@ fn restore_list_worker( let res = proxmox_lang::try_block!({ // phase 0 + // without snapshots and namespace mapping, this is a full restore using several drives + let restore_all = drives.len() > 1 && snapshots.is_empty() && store_map.ns_map.is_none(); let snapshots = if snapshots.is_empty() { let mut restorable = Vec::new(); // restore source namespaces for (store, snapshot) in catalog.list_snapshots() { let (ns, dir) = match parse_ns_and_snapshot(snapshot) { + Ok((ns, dir)) if restore_all && store_map.target_store(store).is_some() => { + (ns, dir) + } Ok((ns, dir)) if store_map.has_full_mapping(store, &ns) => (ns, dir), Err(err) => { warn!("couldn't parse snapshot {snapshot} - {err}"); @@ -671,7 +728,7 @@ fn restore_list_worker( &snapshot, &ns, &dir, - false, + restore_all, &user_info, auth_id, restore_owner, diff --git a/src/bin/proxmox-tape.rs b/src/bin/proxmox-tape.rs index dcbf2d0fb..504e6e380 100644 --- a/src/bin/proxmox-tape.rs +++ b/src/bin/proxmox-tape.rs @@ -906,6 +906,17 @@ async fn backup(mut param: Value) -> Result<(), Error> { schema: DRIVE_NAME_SCHEMA, optional: true, }, + drives: { + description: "List of drives to use. Several drives have to be in the same \ + changer, their media are then read in parallel. Replaces the deprecated \ + 'drive'.", + type: Array, + optional: true, + min_length: 1, + items: { + schema: DRIVE_NAME_SCHEMA, + }, + }, "media-set": { description: "Media set UUID.", type: String, @@ -939,7 +950,9 @@ async fn restore(mut param: Value) -> Result<(), Error> { let (config, _digest) = pbs_config::drive::config()?; - param["drive"] = extract_drive_name(&mut param, &config)?.into(); + if param["drives"].is_null() { + param["drive"] = extract_drive_name(&mut param, &config)?.into(); + } let client = connect_to_localhost()?; -- 2.53.0