From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 0DEC41FF0B6 for ; Thu, 12 Nov 2026 12:41:28 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B219221629; Thu, 12 Nov 2026 12:41:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stuttgart.de; s=s1022; t=1794483670; bh=B719DLrcrUUTc9utF1ChzRvk7AZSZjt3xdwK9yD/mdA=; h=From:To:Cc:Subject:Date:From; b=uXipHJOPxIH8BHUW9NXsPrPAtAZvEd4WhWu2rrppDw/61aiXR6nSwY1s3YrmfU9S2 I6lymtEskSA3zJWhKkJNBSxWJ0rpdsMo2pxHsb014Kb6GnoKl5upuqqFHSuiOVhUdr u5n1OAa4zCzYbsYCTkbgcw307fjJDqKerwTliNTytHtVRumyuXy8aajs1/qNH9G83F W+5EWcZ0ArtutwlOhN1qyF30lx2EUtqTHeiMg5XjpaibpE2AvMsvyiGz3erp3fmEwq z/ZnY3QnWi5jDqolfBxZj5PuxClsOX+mgq94IH7TLVbMehSTIGR8xJ754UnoqigU7Y uK6DvZgPy/kBA== From: Jakob-Tobias Winter To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup 0/4] fix 7333 use multiple tape drives for restoring Date: Thu, 12 Nov 2026 12:37:33 +0100 Message-ID: <20261112113737.748541-1-tobias@stuttgart.de> X-Mailer: git-send-email 2.53.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.037 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: SIO4KM72WIQCEJSSDAOM7XMCPQCQPZCY X-Message-ID-Hash: SIO4KM72WIQCEJSSDAOM7XMCPQCQPZCY X-MailFrom: prvs=f560d3371a=tobias@stuttgart.de X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Jakob-Tobias Winter X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Restoring a media set reads one tape after the other with a single drive, even if the library has more drives that are idle. This series lets a restore use several drives of a changer, so several tapes are read at the same time [0]. Even when the host cannot keep up with the combined data rate, loading, positioning and changer moves on one drive then overlap with reading on the others. This is the follow-up to the RFC [1], thanks to Dominik for the review. Patch 1 adds a pool of drives that runs one job per media. With a single drive, this is the existing loop, so a restore with one drive behaves as before. With several drives, the media in the drives are unloaded first, and the task fails if a needed media is loaded in a drive that is not used. Then each drive runs in its own thread, where it takes the next media, loads it, runs the job and unloads it again. Changer operations are serialized, so two drives do not pick the same free slot when unloading. The first error, or an abort, stops all drives, which then unload their media as well. As the lines of all drives are interleaved in the task log, every message of a drive starts with "drive 'NAME': ". Patch 2 loads the media of the first two phases of the selective restore through the pool, without functional change. Patch 3 adds the 'drives' parameter to the API and to 'proxmox-tape restore'. It replaces 'drive', which still works, but is deprecated. Several drives have to be in the same changer. Every drive is checked for Tape.Read, locked and marked as in use for the whole task. A selective restore then reads the snapshot archives (phase 1) and the chunks (phase 2) on all drives. A full restore writes the snapshot archives straight into the datastore. That is safe with a single drive, because a tape backup writes the chunks of a snapshot before its snapshot archive, to the same or an earlier media. With several drives, a snapshot archive can be read while an earlier media is still being restored, and an aborted restore would leave snapshots with missing chunks behind. So with more than one drive, a full restore runs the phases of the selective restore for all snapshots, which only add the snapshots once all chunks are restored. The restore window already takes that path whenever it sends a namespace mapping. The trade-offs are that the media set catalog is needed, every media is loaded twice (once for the snapshot archives, once for the chunks), the snapshots only show up at the end, and snapshots that already exist in the datastore are skipped with a warning instead of silently. Restoring everything in a single pass, with the snapshots in a temporary directory until all chunks are there, would avoid the second load, as suggested in the review of the RFC. I left that out to keep the changes to the restore logic small, and would look into it as a follow-up. Patch 4 turns the drive selector of the restore window into a multi-select field, which only accepts several drives if they are in the same changer. On abort or error, the existing cleanup of the selective restore removes the temporary directory, so an abort before phase 3 does not add any snapshot. Chunks that were already written stay until garbage collection removes them, as today. Changes since the RFC [1]: - 'drives' replaces 'drive', instead of adding 'additional-drives', and there is no limit for the number of drives anymore - several drives have to be in the same changer, the API and the restore window check that - the drives are unloaded first, a needed media in a drive that is not used is an error, and each drive takes the next media in order and unloads it after use, instead of preferring the media loaded in it - a failing drive aborts the task, instead of using a stop flag and a task context of its own, and a poisoned lock is not ignored anymore - the pool takes the maps of the two restore phases as they are and looks up the media labels in the inventory itself - one multi-select field in the restore window instead of a second field for additional drives - rebased on a newer master Testing: I built this version on top of 4.2.8 and tested it with several drives on a tape library with LTO-7 media, where four of the drives are available to PBS: restoring a whole media set, restoring a selected snapshot, aborting a restore, after which the drives unload their media, and the error for a needed media that is loaded in a drive that is not used. I also tested the unloading of drives that hold a media when the restore starts, and the restore window. In addition, with real daemons and four virtual tape drives, each with a copy of the tapes: with one drive, the task logs of a full restore, a restore of selected snapshots and one with a namespace mapping are the same as on master. With two to four drives, the same three kinds of restore give datastores with the content of the source, which also verify and read back correctly. A drive that fails stops the others and unloads its media, and the task ends with its error, without adding a snapshot. Drives that are not in the same changer are refused. The test suite passes and clippy shows no new warnings. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7333 [1] https://lore.proxmox.com/pbs-devel/20261106133826.1692620-2-tobias@stuttgart.de/ Jakob-Tobias Winter (4): tape: drive: add a pool to use several drives for one task tape: restore: use a drive pool for the selective restore close #7333: tape: restore: allow using several drives in parallel ui: tape restore: allow selecting several drives src/api2/tape/restore.rs | 180 ++++++++++++++++------- src/bin/proxmox-tape.rs | 15 +- src/tape/drive/mod.rs | 40 ++++- src/tape/drive/pool.rs | 259 +++++++++++++++++++++++++++++++++ www/tape/form/DriveSelector.js | 17 +++ www/tape/window/TapeRestore.js | 5 +- 6 files changed, 455 insertions(+), 61 deletions(-) create mode 100644 src/tape/drive/pool.rs base-commit: 68577e65400958d9f7e76a7722669fb359b69d4b -- 2.53.0