[pve-devel] [PATCH qemu] PVE: fix aborting multiple 'CREATED' jobs in sequential transaction

[pve-devel] [PATCH qemu] PVE: fix aborting multiple 'CREATED' jobs in sequential transaction

[Stefan Reiter]

Deadlocks could occur in the AIO_WAIT_WHILE loop in job_finish_sync,
which would wait for CREATED but not running jobs to complete, even
though job_enter is a no-op in that scenario. Mark offending jobs as
ABORTING immediately via job_update_rc if required.

Manifested itself in cancelling or failing backups with more than 2
drives.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---

Potential fix for #3225 and related forum threads:
https://forum.proxmox.com/threads/problem-mit-backup.80418/
https://forum.proxmox.com/threads/vm-hard-freezes-on-backup.81752/

 job.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/job.c b/job.c
index 97ee97a192..51984e557c 100644
--- a/job.c
+++ b/job.c
@@ -1035,6 +1035,13 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)
         return -EBUSY;
     }
 
+    /* in a sequential transaction jobs with status CREATED can appear at time
+     * of cancelling, these have not begun work so job_enter won't do anything,
+     * let's ensure they are marked as ABORTING if required */
+    if (job->status == JOB_STATUS_CREATED && job->txn->sequential) {
+        job_update_rc(job);
+    }
+
     AIO_WAIT_WHILE(job->aio_context,
                    (job_enter(job), !job_is_completed(job)));
 
-- 
2.20.1

Re: [pve-devel] [PATCH qemu] PVE: fix aborting multiple 'CREATED' jobs in sequential transaction

[Mira Limbeck]

Tested Stefan's prebuilt qemu package with this patch applied and my VM 
that has the issue.

Additionally tested the case of a full backup target 
(https://forum.proxmox.com/threads/vm-hard-freezes-on-backup.81752/) and 
it no longer hangs.

Looks good in my tests, so:

Tested-by: Mira Limbeck <m.limbeck@proxmox.com>

On 1/4/21 2:49 PM, Stefan Reiter wrote:
> Deadlocks could occur in the AIO_WAIT_WHILE loop in job_finish_sync,
> which would wait for CREATED but not running jobs to complete, even
> though job_enter is a no-op in that scenario. Mark offending jobs as
> ABORTING immediately via job_update_rc if required.
>
> Manifested itself in cancelling or failing backups with more than 2
> drives.
>
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
>
> Potential fix for #3225 and related forum threads:
> https://forum.proxmox.com/threads/problem-mit-backup.80418/
> https://forum.proxmox.com/threads/vm-hard-freezes-on-backup.81752/
>
>   job.c | 7 +++++++
>   1 file changed, 7 insertions(+)
>
> diff --git a/job.c b/job.c
> index 97ee97a192..51984e557c 100644
> --- a/job.c
> +++ b/job.c
> @@ -1035,6 +1035,13 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)
>           return -EBUSY;
>       }
>   
> +    /* in a sequential transaction jobs with status CREATED can appear at time
> +     * of cancelling, these have not begun work so job_enter won't do anything,
> +     * let's ensure they are marked as ABORTING if required */
> +    if (job->status == JOB_STATUS_CREATED && job->txn->sequential) {
> +        job_update_rc(job);
> +    }
> +
>       AIO_WAIT_WHILE(job->aio_context,
>                      (job_enter(job), !job_is_completed(job)));
>   

[pve-devel] applied: [PATCH qemu] PVE: fix aborting multiple 'CREATED' jobs in sequential transaction

[Wolfgang Bumiller]

applied

On Mon, Jan 04, 2021 at 02:49:14PM +0100, Stefan Reiter wrote:
> Deadlocks could occur in the AIO_WAIT_WHILE loop in job_finish_sync,
> which would wait for CREATED but not running jobs to complete, even
> though job_enter is a no-op in that scenario. Mark offending jobs as
> ABORTING immediately via job_update_rc if required.
> 
> Manifested itself in cancelling or failing backups with more than 2
> drives.
> 
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
> 
> Potential fix for #3225 and related forum threads:
> https://forum.proxmox.com/threads/problem-mit-backup.80418/
> https://forum.proxmox.com/threads/vm-hard-freezes-on-backup.81752/
> 
>  job.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/job.c b/job.c
> index 97ee97a192..51984e557c 100644
> --- a/job.c
> +++ b/job.c
> @@ -1035,6 +1035,13 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)
>          return -EBUSY;
>      }
>  
> +    /* in a sequential transaction jobs with status CREATED can appear at time
> +     * of cancelling, these have not begun work so job_enter won't do anything,
> +     * let's ensure they are marked as ABORTING if required */
> +    if (job->status == JOB_STATUS_CREATED && job->txn->sequential) {
> +        job_update_rc(job);
> +    }
> +
>      AIO_WAIT_WHILE(job->aio_context,
>                     (job_enter(job), !job_is_completed(job)));
>  
> -- 
> 2.20.1