[pdm-devel] [PATCH proxmox-datacenter-manager 0/2] improve handling of missing task cache files

[pdm-devel] [PATCH proxmox-datacenter-manager 0/2] improve handling of missing task cache files

[Lukas Wagner]

This avoids misleading 'could not create task iterator' errors in some
cases.

Lukas Wagner (2):
  server: remote task archive: handle missing journal file gracefully
  server: remote task cache: improve handling of missing archive files

 server/src/remote_tasks/task_cache.rs | 91 +++++++++++++++------------
 1 file changed, 51 insertions(+), 40 deletions(-)

-- 
2.47.2



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH proxmox-datacenter-manager 1/2] server: remote task archive: handle missing journal file gracefully

[Lukas Wagner]

This file is only created after the first update round. This fixes a
'could not create task archive iterator' error when requesting the task
list directly after server startup, in case the journal was deleted or 
has not existed before.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 server/src/remote_tasks/task_cache.rs | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/server/src/remote_tasks/task_cache.rs b/server/src/remote_tasks/task_cache.rs
index e9e708e4..57047c8c 100644
--- a/server/src/remote_tasks/task_cache.rs
+++ b/server/src/remote_tasks/task_cache.rs
@@ -952,14 +952,27 @@ impl<'a> TaskArchiveIterator<'a> {
             .peekable();
 
         if let Some(journal) = journal {
-            let journal_reader = Box::new(BufReader::new(File::open(journal)?));
-            let journal_task_iterator = JournalIterator::new(journal_reader).peekable();
-            let merge_task_iter = MergeTaskIterator::new(journal_task_iterator, inner);
+            let file = File::open(journal);
 
-            Ok(Self {
-                inner: Box::new(merge_task_iter),
-                _lock: lock,
-            })
+            match file {
+                Ok(file) => {
+                    let journal_reader = Box::new(BufReader::new(file));
+                    let journal_task_iterator = JournalIterator::new(journal_reader).peekable();
+                    let merge_task_iter = MergeTaskIterator::new(journal_task_iterator, inner);
+
+                    Ok(Self {
+                        inner: Box::new(merge_task_iter),
+                        _lock: lock,
+                    })
+                }
+                Err(err) if err.kind() == ErrorKind::NotFound => Ok(Self {
+                    inner: Box::new(inner),
+                    _lock: lock,
+                }),
+                Err(err) => {
+                    return Err(err.into());
+                }
+            }
         } else {
             Ok(Self {
                 inner: Box::new(inner),
-- 
2.47.2



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH proxmox-datacenter-manager 2/2] server: remote task cache: improve handling of missing archive files

[Lukas Wagner]

This avoids misleading 'could not create task iterator' errors in some
cases, for instance when requesting a task list when the 'active' is
missing/has not been created yet.

The changed handling allows us to skip creating the 'active' file in
'init', it will be created when needed.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 server/src/remote_tasks/task_cache.rs | 64 +++++++++++++--------------
 1 file changed, 31 insertions(+), 33 deletions(-)

diff --git a/server/src/remote_tasks/task_cache.rs b/server/src/remote_tasks/task_cache.rs
index 57047c8c..aa038443 100644
--- a/server/src/remote_tasks/task_cache.rs
+++ b/server/src/remote_tasks/task_cache.rs
@@ -214,19 +214,6 @@ impl WritableTaskCache {
     ///
     /// This function only has an effect if there are no archive files yet.
     pub fn init(&self, now: i64) -> Result<(), Error> {
-        let active_filename = self.cache.base_path.join(ACTIVE_FILENAME);
-
-        if !active_filename.exists() {
-            let mut file = OpenOptions::new()
-                .create(true)
-                .write(true)
-                .open(&active_filename)?;
-
-            self.cache
-                .create_options
-                .apply_to(&mut file, &active_filename)?;
-        }
-
         if self.cache.archive_files(&self.lock)?.is_empty() {
             for i in 0..self.cache.max_files {
                 self.new_file(
@@ -694,6 +681,12 @@ impl WritableTaskCache {
 
         let archive_iter = file
             .iter()?
+            .with_context(|| {
+                format!(
+                    "task archive file '{}' disappeared while merging tasks",
+                    file.path.display()
+                )
+            })?
             .flat_map(|item| match item {
                 Ok(item) => Some(item),
                 Err(err) => {
@@ -819,33 +812,28 @@ impl TaskCache {
         lock: &'a TaskCacheLock,
     ) -> Result<TaskArchiveIterator<'a>, Error> {
         let journal_file = self.base_path.join(WAL_FILENAME);
-        let active_path = self.base_path.join(ACTIVE_FILENAME);
 
         match mode {
             GetTasks::All => {
                 let mut archive_files = self.archive_files(lock)?;
                 archive_files.reverse();
 
-                if active_path.exists() {
-                    archive_files.push(ArchiveFile {
-                        path: self.base_path.join(ACTIVE_FILENAME),
-                        compressed: false,
-                        starttime: 0,
-                    });
-                }
+                archive_files.push(ArchiveFile {
+                    path: self.base_path.join(ACTIVE_FILENAME),
+                    compressed: false,
+                    starttime: 0,
+                });
 
                 TaskArchiveIterator::new(Some(journal_file), archive_files, lock)
             }
             GetTasks::Active => {
                 let mut archive_files = Vec::new();
 
-                if active_path.exists() {
-                    archive_files.push(ArchiveFile {
-                        path: self.base_path.join(ACTIVE_FILENAME),
-                        compressed: false,
-                        starttime: 0,
-                    });
-                }
+                archive_files.push(ArchiveFile {
+                    path: self.base_path.join(ACTIVE_FILENAME),
+                    compressed: false,
+                    starttime: 0,
+                });
 
                 TaskArchiveIterator::new(None, archive_files, lock)
             }
@@ -1026,10 +1014,13 @@ impl Iterator for InnerTaskArchiveIterator {
                     let next_file = self.files.pop()?;
 
                     match next_file.iter() {
-                        Ok(iter) => {
+                        Ok(Some(iter)) => {
                             self.current = Some(iter);
                             break 'inner;
                         }
+                        Ok(None) => {
+                            // File does not exist, nothing to log in this case.
+                        }
                         Err(err) => {
                             log::error!("could not create archive iterator while iteration over task archive files, skipping: {err:#}")
                         }
@@ -1053,9 +1044,16 @@ struct ArchiveFile {
 
 impl ArchiveFile {
     /// Create an [`ArchiveIterator`] for this file.
-    fn iter(&self) -> Result<ArchiveIterator, Error> {
-        let fd = File::open(&self.path)
-            .with_context(|| format!("failed to open archive file {}", self.path.display()))?;
+    fn iter(&self) -> Result<Option<ArchiveIterator>, Error> {
+        let fd = match File::open(&self.path) {
+            Ok(fd) => fd,
+            Err(err) if err.kind() == ErrorKind::NotFound => {
+                return Ok(None);
+            }
+            Err(err) => {
+                return Err(err.into());
+            }
+        };
 
         let iter = if self.compressed {
             let reader = zstd::stream::read::Decoder::new(fd).with_context(|| {
@@ -1069,7 +1067,7 @@ impl ArchiveFile {
             ArchiveIterator::new(Box::new(BufReader::new(fd)))
         };
 
-        Ok(iter)
+        Ok(Some(iter))
     }
 
     fn compress(&mut self, options: CreateOptions) -> Result<(), Error> {
-- 
2.47.2



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] applied-series: [PATCH proxmox-datacenter-manager 0/2] improve handling of missing task cache files

[Thomas Lamprecht]

On Wed, 03 Sep 2025 16:48:50 +0200, Lukas Wagner wrote:
> This avoids misleading 'could not create task iterator' errors in some
> cases.
> 
> Lukas Wagner (2):
>   server: remote task archive: handle missing journal file gracefully
>   server: remote task cache: improve handling of missing archive files
> 
> [...]

Applied, thanks!

[1/2] server: remote task archive: handle missing journal file gracefully
      commit: 1b6dee21faf38aecebc1e58b664bd9fe4b02be3e
[2/2] server: remote task cache: improve handling of missing archive files
      commit: 501d68406013d045176ebfb22b40371994d6ebdc


_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel