From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Dominik Csapak <d.csapak@proxmox.com>,
Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>,
Thomas Lamprecht <t.lamprecht@proxmox.com>
Subject: Re: [pbs-devel] [PATCH proxmox 2/2] rest-server: close race window when updating worker task count
Date: Mon, 02 Dec 2024 10:14:44 +0100 [thread overview]
Message-ID: <1733130843.g4mnhh0hy8.astroid@yuna.none> (raw)
In-Reply-To: <211810f3-3eef-42bf-b17d-6f8f5f24c8a8@proxmox.com>
On November 29, 2024 3:20 pm, Dominik Csapak wrote:
> On 11/29/24 14:27, Thomas Lamprecht wrote:
>> Am 29.11.24 um 14:13 schrieb Fabian Grünbichler:
>>> this mimics how the count is updated when spawning a new task - the lock scope
>>> needs to cover the count update itself, else there's a race when multiple
>>> worker's log their result at the same time..
>>>
>>> Co-developed-by: Dominik Csapak <d.csapak@proxmox.com>
>>> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
>>> ---
>>> proxmox-rest-server/src/worker_task.rs | 3 ++-
>>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
>>> index 3ca93965..018d18c0 100644
>>> --- a/proxmox-rest-server/src/worker_task.rs
>>> +++ b/proxmox-rest-server/src/worker_task.rs
>>> @@ -1023,7 +1023,8 @@ impl WorkerTask {
>>>
>>> WORKER_TASK_LIST.lock().unwrap().remove(&self.upid.task_id);
>>> let _ = self.setup.update_active_workers(None);
>>> - set_worker_count(WORKER_TASK_LIST.lock().unwrap().len());
>>> + let lock = WORKER_TASK_LIST.lock().unwrap();
>>
>> why not use this also for the remove operation above? I.e. something like:
>>
>> let locked_worker_tasks = WORKER_TASK_LIST.lock().unwrap();
>>
>> locked_worker_tasks.remove(&self.upid.task_id);
>>
>> set_worker_count(locked_worker_tasks.len())
>>
>> If there are technical reason speaking against this, which I hope not, then a
>> comment would be definitively warranted, otherwise using a single lock would
>> IMO make this a bit clearer and locking twice isn't exactly cheaper.
>
> here the reason of the split lock is that the 'self.setup.update_active_workers` internally
> can take a lock to the WORKER_TASK_LIST, so we can't hold one over that call
>
> not super sure if can reorder these, so that we reduce the count before updating
> though. From what i understand though we want to remove ourselves from the list
> of actives tasks before reducing that counter.
>
> as fabian indicated in the other patch, we should probably split up
> the 'update_active_workers' into seperate methods to
> * add one worker
> * remove one worker
> * housekeeping for leftover workers
>
> then we could design the removal in a way that does not rely on the WORKER_TASK_LIST
> in the first place thus we could remove it from the active list before removing it
> from the internal hashmap (and could take a lock around both, the list and the count)
yes to all of the above. and a comment why the lock is obtained twice
probably is a good ideal for the stop-gap fix.
>
>>
>> Looks OK besides that, but would still want to take a closer look.
>>
>>> + set_worker_count(lock.len());
>>> }
>>>
>>> /// Log a message.
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
next prev parent reply other threads:[~2024-12-02 9:15 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-11-29 13:13 [pbs-devel] [RFC proxmox 0/2] worker task setup improvements Fabian Grünbichler
2024-11-29 13:13 ` [pbs-devel] [PATCH proxmox 1/2] rest-server: handle failure in worker task setup correctly Fabian Grünbichler
2024-11-29 13:34 ` Thomas Lamprecht
2024-12-02 9:14 ` Fabian Grünbichler
2024-11-29 13:13 ` [pbs-devel] [PATCH proxmox 2/2] rest-server: close race window when updating worker task count Fabian Grünbichler
2024-11-29 13:27 ` Thomas Lamprecht
2024-11-29 14:20 ` Dominik Csapak
2024-12-02 9:14 ` Fabian Grünbichler [this message]
2024-11-29 14:53 ` [pbs-devel] [RFC proxmox 0/2] worker task setup improvements Dominik Csapak
2024-12-02 13:04 ` [pbs-devel] superseded: " Fabian Grünbichler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1733130843.g4mnhh0hy8.astroid@yuna.none \
--to=f.gruenbichler@proxmox.com \
--cc=d.csapak@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
--cc=t.lamprecht@proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox