From: Wolfgang Bumiller <w.bumiller@proxmox.com>
To: Lukas Wagner <l.wagner@proxmox.com>
Cc: pdm-devel@lists.proxmox.com
Subject: Re: [pdm-devel] [PATCH proxmox-datacenter-manager v3 3/6] remote tasks: improve locking for task archive iterator
Date: Fri, 18 Apr 2025 09:12:39 +0200 [thread overview]
Message-ID: <onhaulwwxv7psfm37qtwadpogx5jvjqxwgi6xehub3majun5m6@ruadm36gdazo> (raw)
In-Reply-To: <20250417132256.282624-4-l.wagner@proxmox.com>
On Thu, Apr 17, 2025 at 03:22:53PM +0200, Lukas Wagner wrote:
> Instead of awkwardly using into_lock for keeping the archive locked,
> pass the lock as a reference to TaskCache::get_tasks_with_lock. The
> iterator itself only holds the lock if get use the auto-locking
> TaskCache::get_tasks, otherwise we ensure that the lock lives long
> enough via lifetimes and PhantomData in the iterator.
>
> Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
> ---
>
> Notes:
> New in v3
>
> server/src/remote_tasks/mod.rs | 2 +-
> server/src/remote_tasks/task_cache.rs | 62 +++++++++++++--------------
> 2 files changed, 31 insertions(+), 33 deletions(-)
>
> diff --git a/server/src/remote_tasks/mod.rs b/server/src/remote_tasks/mod.rs
> index 126c9ad3..b0fc052f 100644
> --- a/server/src/remote_tasks/mod.rs
> +++ b/server/src/remote_tasks/mod.rs
> @@ -24,7 +24,7 @@ pub async fn get_tasks(filters: TaskFilters) -> Result<Vec<TaskListItem>, Error>
> GetTasks::All
> };
>
> - for task in &mut cache
> + for task in cache
> .get_tasks(which)?
> .skip(filters.start as usize)
> .take(filters.limit as usize)
> diff --git a/server/src/remote_tasks/task_cache.rs b/server/src/remote_tasks/task_cache.rs
> index f847d441..c59c5235 100644
> --- a/server/src/remote_tasks/task_cache.rs
> +++ b/server/src/remote_tasks/task_cache.rs
> @@ -6,6 +6,7 @@ use std::{
> fs::File,
> io::{BufRead, BufReader, BufWriter, ErrorKind, Lines, Write},
> iter::Peekable,
> + marker::PhantomData,
> path::{Path, PathBuf},
> time::Duration,
> };
> @@ -213,8 +214,8 @@ impl TaskCache {
> .map(|(remote, add_tasks)| (remote, add_tasks.update_most_recent_archive_timestamp))
> .collect();
>
> - let mut task_iter = self
> - .get_tasks_with_lock(GetTasks::Active, lock)
> + let task_iter = self
> + .get_tasks_with_lock(GetTasks::Active, &lock)
> .context("failed to create archive iterator for active tasks")?;
>
> let mut active_tasks =
> @@ -226,10 +227,6 @@ impl TaskCache {
> }
> }));
>
> - // Consume the iterator to get back the lock. The lock is held
> - // until _lock is finally dropped at the end of the function.
> - let _lock = task_iter.into_lock();
> -
> let mut new_finished_tasks = Vec::new();
>
> for task in tasks {
> @@ -387,7 +384,7 @@ impl TaskCache {
>
> let mut tasks = Vec::new();
> let mut task_iter = self
> - .get_tasks_with_lock(GetTasks::Active, lock)
> + .get_tasks_with_lock(GetTasks::Active, &lock)
> .context("failed to create active task iterator")?;
>
> for task in &mut task_iter {
> @@ -404,8 +401,6 @@ impl TaskCache {
> tasks.push(task.clone());
> tasks.sort_by(compare_tasks_reverse);
>
> - let _lock = task_iter.into_lock();
> -
> let mut state = self.read_state();
>
> state
> @@ -432,24 +427,28 @@ impl TaskCache {
> ///
> /// This function will request a non-exclusive read-lock, don't call if
> /// you already hold a lock for this cache. See [`Self::get_tasks_with_lock`].
> - pub fn get_tasks(&self, mode: GetTasks) -> Result<TaskArchiveIterator, Error> {
> - let lock = self
> - .lock(false)
> - .context("get_tasks: failed to acquire lock")?;
> - self.get_tasks_with_lock(mode, lock)
> + pub fn get_tasks(&self, mode: GetTasks) -> Result<TaskArchiveIterator<'static>, Error> {
> + let lock = self.lock(false).context("failed to lock archive")?;
> + self.get_tasks_impl(mode, Some(lock))
> .context("failed to create task archive iterator")
> }
>
> /// Iterate over cached tasks.
> ///
> - /// This function requires you to pass a lock. If you want to continue to hold the lock
> - /// after iterating, you can consume the iterator by calling
> - /// [`TaskArchiveIterator::into_lock`], yielding the original lock.
> - pub fn get_tasks_with_lock(
> + /// This function requires you to pass a lock.
> + pub fn get_tasks_with_lock<'a>(
> &self,
> mode: GetTasks,
> - lock: TaskCacheLock,
> - ) -> Result<TaskArchiveIterator, Error> {
> + _lock: &'a TaskCacheLock,
> + ) -> Result<TaskArchiveIterator<'a>, Error> {
> + self.get_tasks_impl(mode, None)
> + }
> +
> + pub fn get_tasks_impl<'a>(
^ This must not be `pub`, given the lifetime is chosen "at will" by the
caller and the relation to the lock's lifetime is not compile-time
enforced. Only the non-`_impl` functions enforce this.
> + &self,
> + mode: GetTasks,
> + lock: Option<TaskCacheLock>,
> + ) -> Result<TaskArchiveIterator<'a>, Error> {
> match mode {
> GetTasks::All => {
> let archive_files = self.archive_files()?.into_iter().map(|pair| pair.file);
> @@ -682,32 +681,31 @@ pub fn compare_tasks_reverse(a: &TaskCacheItem, b: &TaskCacheItem) -> Ordering {
> }
>
> /// Iterator over the task archive.
> -pub struct TaskArchiveIterator {
> +pub struct TaskArchiveIterator<'a> {
> /// Archive files to read.
> files: Box<dyn Iterator<Item = PathBuf>>,
> /// Archive iterator we are currently using, if any
> current: Option<ArchiveIterator<BufReader<File>>>,
> - /// Lock for this archive.
> - lock: TaskCacheLock,
> + /// Lock for this archive. This contains the lock in case we
> + /// need to keep the archive locked while iterating over it.
> + _lock: Option<TaskCacheLock>,
> + /// PhantomData to bind the lifetime of the iterator to an externally held lock.
> + _lifetime: PhantomData<&'a ()>,
> }
>
> -impl TaskArchiveIterator {
> +impl TaskArchiveIterator<'_> {
> /// Create a new task archive iterator.
> - pub fn new(files: Box<dyn Iterator<Item = PathBuf>>, lock: TaskCacheLock) -> Self {
> + pub fn new(files: Box<dyn Iterator<Item = PathBuf>>, lock: Option<TaskCacheLock>) -> Self {
> Self {
> files,
> current: None,
> - lock,
> + _lock: lock,
> + _lifetime: PhantomData,
> }
> }
> -
> - /// Return the task archive lock, consuming `self`.
> - pub fn into_lock(self) -> TaskCacheLock {
> - self.lock
> - }
> }
>
> -impl Iterator for &mut TaskArchiveIterator {
> +impl Iterator for TaskArchiveIterator<'_> {
> type Item = Result<TaskCacheItem, Error>;
>
> fn next(&mut self) -> Option<Self::Item> {
> --
> 2.39.5
_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel
next prev parent reply other threads:[~2025-04-18 7:12 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-04-17 13:22 [pdm-devel] [PATCH proxmox-datacenter-manager v3 0/6] remote task cache fetching task / better cache backend Lukas Wagner
2025-04-17 13:22 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 1/6] remote tasks: implement improved cache for remote tasks Lukas Wagner
2025-04-17 13:22 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 2/6] remote tasks: add background task for task polling, use new task cache Lukas Wagner
2025-04-17 13:22 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 3/6] remote tasks: improve locking for task archive iterator Lukas Wagner
2025-04-18 7:12 ` Wolfgang Bumiller [this message]
2025-04-17 13:22 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 4/6] pdm-api-types: remote tasks: add new_from_str constructor for TaskStateType Lukas Wagner
2025-04-17 13:22 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 5/6] fake remote: make the fake_remote feature compile again Lukas Wagner
2025-04-17 13:22 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 6/6] fake remote: clippy fixes Lukas Wagner
2025-04-17 15:31 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 0/6] remote task cache fetching task / better cache backend Thomas Lamprecht
2025-04-18 7:12 ` Wolfgang Bumiller
2025-04-18 7:24 ` Lukas Wagner
2025-04-18 8:08 ` Lukas Wagner
2025-04-18 8:33 ` [pdm-devel] superseded: " Lukas Wagner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=onhaulwwxv7psfm37qtwadpogx5jvjqxwgi6xehub3majun5m6@ruadm36gdazo \
--to=w.bumiller@proxmox.com \
--cc=l.wagner@proxmox.com \
--cc=pdm-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal