public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH proxmox v2 0/4] worker task setup improvements
Date: Mon,  2 Dec 2024 14:04:08 +0100	[thread overview]
Message-ID: <20241202130412.1290616-1-f.gruenbichler@proxmox.com> (raw)

This series fixes two issues related to reset server shutdown/reload and
worker task accounting.

issue 1 (patches 1, rfcs 3, 4):

if WorkerTask::new returned an error because updating the task indices
failed (for example, because the lock covering such updates was not
acquired because of a timeout), the task was registered in the worker
task list, but not returned to the caller, which means the task could
never actually execute and reach its cleanup/log_result stage, which
would unregister it again. effectively, in such a scenario the worker
task is "leaked", but the task count decrement never happens, which in
turn means the corresponding proxy can never shutdown, since it will
wait for the phantom task to finish forever.

this issue was actually found in the wild on a system with lots of
activity.

issue 2 (patch 2):

a lock scope issue could cause a temporary inconsistency between the
task list and task count, if multiple tasks log their result in
parallel. the discrepancy disappars with the next task that is created
or logs its result, since the count is always reset to the current count
and not incremented/decremented.

this issue was found while analyzing the code.

Fabian Grünbichler (4):
  rest-server: handle failure in worker task setup correctly
  rest-server: close race window when updating worker task count
  rest-server: make worker task creation error handling more idiomatic
  rest-server: increase task index lock timeout to 15s

 proxmox-rest-server/src/worker_task.rs | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

             reply	other threads:[~2024-12-02 13:04 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-12-02 13:04 Fabian Grünbichler [this message]
2024-12-02 13:04 ` [pbs-devel] [PATCH proxmox v2 1/4] rest-server: handle failure in worker task setup correctly Fabian Grünbichler
2024-12-02 15:57   ` [pbs-devel] applied: " Thomas Lamprecht
2024-12-02 13:04 ` [pbs-devel] [PATCH proxmox v2 2/4] rest-server: close race window when updating worker task count Fabian Grünbichler
2024-12-02 15:57   ` [pbs-devel] applied: " Thomas Lamprecht
2024-12-02 13:04 ` [pbs-devel] [RFC proxmox v2 3/4] rest-server: make worker task creation error handling more idiomatic Fabian Grünbichler
2024-12-02 15:58   ` Thomas Lamprecht
2024-12-02 13:04 ` [pbs-devel] [RFC proxmox v2 4/4] rest-server: increase task index lock timeout to 15s Fabian Grünbichler
2024-12-02 15:59   ` [pbs-devel] applied: " Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20241202130412.1290616-1-f.gruenbichler@proxmox.com \
    --to=f.gruenbichler@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal