[pbs-devel] [PATCH proxmox-backup] tape: wait for calibration of LTO-9 tapes

[pbs-devel] [PATCH proxmox-backup] tape: wait for calibration of LTO-9 tapes

[Dominik Csapak]

Since LTO-9, initial loading of tapes into a drive can block up to 2
hours according to the spec. In case we run into a ready check timeout,
query the drive, and increase the timeout to 2 hours and 5 minutes if
it's calibrating (5 minutes headroom).

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 pbs-tape/src/sg_tape.rs | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/pbs-tape/src/sg_tape.rs b/pbs-tape/src/sg_tape.rs
index 915dee28a..3ce144ad5 100644
--- a/pbs-tape/src/sg_tape.rs
+++ b/pbs-tape/src/sg_tape.rs
@@ -662,7 +662,8 @@ impl SgTape {
     pub fn wait_until_ready(&mut self, timeout: Option<u64>) -> Result<(), Error> {
         let start = SystemTime::now();
         let timeout = timeout.unwrap_or(Self::SCSI_TAPE_DEFAULT_TIMEOUT as u64);
-        let max_wait = std::time::Duration::new(timeout, 0);
+        let mut max_wait = std::time::Duration::new(timeout, 0);
+        let mut increased_timeout = false;
 
         loop {
             match self.test_unit_ready() {
@@ -670,6 +671,16 @@ impl SgTape {
                 _ => {
                     std::thread::sleep(std::time::Duration::new(1, 0));
                     if start.elapsed()? > max_wait {
+                        if !increased_timeout {
+                            if let Ok(DeviceActivity::Calibrating) =
+                                read_device_activity(&mut self.file)
+                            {
+                                log::info!("Detected drive calibration, increasing timeout to 2 hours 5 minutes");
+                                max_wait = std::time::Duration::new(2 * 60 * 60 + 5 * 60, 0);
+                                increased_timeout = true;
+                                continue;
+                            }
+                        }
                         bail!("wait_until_ready failed - got timeout");
                     }
                 }
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup] tape: wait for calibration of LTO-9 tapes

[Thomas Lamprecht]

(re-send with list in CC, sorry)

On 15/04/2025 13:40, Dominik Csapak wrote:
> Since LTO-9, initial loading of tapes into a drive can block up to 2
> hours according to the spec. In case we run into a ready check timeout,
> query the drive, and increase the timeout to 2 hours and 5 minutes if
> it's calibrating (5 minutes headroom).

Is there any (spec) reference we can link here?

And the 2h5m would be on top of the previous max_wait, AFAICT.

And, didn't we already fix something like this? Or at least had some
other handling added for long initial initialisation of LTO9 tapes, or
was that something different or just a discussion without a patch.
Not really important though, just wondering.

Changes look alright besides of those meta things, so I could fleece
any relevant info into the commit message too on applying, if nothing
else comes up.



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup] tape: wait for calibration of LTO-9 tapes

[Dominik Csapak]

On 4/15/25 17:51, Thomas Lamprecht wrote:
> (re-send with list in CC, sorry)
> 
> On 15/04/2025 13:40, Dominik Csapak wrote:
>> Since LTO-9, initial loading of tapes into a drive can block up to 2
>> hours according to the spec. In case we run into a ready check timeout,
>> query the drive, and increase the timeout to 2 hours and 5 minutes if
>> it's calibrating (5 minutes headroom).
> 
> Is there any (spec) reference we can link here?

To my knowledge the LTO spec is not published in a standard format like
e.g. other RFC or such, but one can find the IBM and HP LTO
SCSI references rather easily [0][1]

as for the timeout, IBM says it only in their recommendations:
 > Although most optimizations will complete within 60 minutes some optimizations may take up to 2 
hours.

and HP:
 > Media initialization adds a variable amount of time to the initialization process that typically 
takes between 20 minutes and 2 hours.

So it seems there not a hard limit and depends...
In my tests it always took around the 1 hour mark (on available hardware here)


> 
> And the 2h5m would be on top of the previous max_wait, AFAICT.
> 

no actually. i overwrite the old 'max_wait' variable and we calculate
from the same start point (with start.elapsed()) we only check if that
is greater than the timeout, so I increase *to* 2h5m

> And, didn't we already fix something like this? Or at least had some
> other handling added for long initial initialisation of LTO9 tapes, or
> was that something different or just a discussion without a patch.
> Not really important though, just wondering.

Sorry I should have added more context to the commit message. We did in fact
increase the timeout, but only for long formatting, and put a section
in the docs explaining the initialization and that it's recommended
to do that beforehand with the vendor tools (IMHO it still is
even with this patch)

Still, forum users encountered timeout issues e.g. when using
barcode labeling and not initializing beforehand, so I thought
adding a general handling for that in the 'wait_until_ready'
(where most of the timeouts from this will occur) makes sense
to decrease the friction of the UX (even if it takes longer to finish).

> 
> Changes look alright besides of those meta things, so I could fleece
> any relevant info into the commit message too on applying, if nothing
> else comes up.
> 

thanks!


0: IBM LTO-9 
https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference_GA32-0928-05%20(EXTERNAL)_0.pdf
1: HP LTO-9 
https://support.hpe.com/hpesc/public/docDisplay?docId=sd00001239en_us&page=GUID-D7147C7F-2016-0901-0921-000000000450.html


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] applied: [PATCH proxmox-backup] tape: wait for calibration of LTO-9 tapes

[Thomas Lamprecht]

On Tue, 15 Apr 2025 13:40:43 +0200, Dominik Csapak wrote:
> Since LTO-9, initial loading of tapes into a drive can block up to 2
> hours according to the spec. In case we run into a ready check timeout,
> query the drive, and increase the timeout to 2 hours and 5 minutes if
> it's calibrating (5 minutes headroom).
> 
> 

Applied, thanks! I extended the commit message with the follow-up info you
provided and also referenced the commit that implemented the longer
timeout for the format procedure.

btw. there is a theoretical case where one could call this method with a
timeout greater than your fallback and the tape needing even longer than
normal, in that case we would do an extra checking round before decreasing
the timeout and then bail on the next round, so not even problematic, just
might be a bit odd on debugging in the very unlikely case this happens in
the first place. That said, a doc-comment for the method and maybe even a
check if the original timeout parameter is already longer than 2h5m would
not exactly hurt and clarify this. But if that makes even sense it really
can be done as a follow up.

[1/1] tape: wait for calibration of LTO-9 tapes
      commit: 07a21616c28a09efb6039e79163b9c4ac8610565


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel