From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 293BE1FF0B3 for ; Wed, 11 Nov 2026 22:58:25 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C802621631; Wed, 11 Nov 2026 22:58:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794434290; x=1795039090; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=ysycfg2Ajvos/dziERw5zB3S9cR7e3Teh7O4IpvSNLE=; b=VqtuE1fAn0mgZkfdB26LKab0KuaR+EL9LUsMpDZ6mGp4XMm1CZA1Tq4AvdeS+R1PvZ pO6KWbLb177IaHxz3xlYLWLgHhk8AJf2XtAF6GoHjpWbeKohV3YX6isAvrgiQeGoHQWw ybxQPXo2ZW1D5Dsd5US1XR6yXuyfVvQO1E2uUu+ufCrmRUBMaZ/J045LCjkDEnDOFVeT RRaUjBfnWMeW87X3ecEX/caCMATGd0C8US7TtIkaqZ4VuSRkiZdussfeHU4JBJcxM7Pc noPBOHRRUVIInnsVcRQjl0F63a0Hq3D09NvWP3DX6hSlRNTr2e7Ntr0E0TWNU/Cwil1T v2fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794434290; x=1795039090; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to:content-type; bh=ysycfg2Ajvos/dziERw5zB3S9cR7e3Teh7O4IpvSNLE=; b=EEVmNMI7/SZtErwZUfJFL9aybW/CIueKheyaqhhuF+X8vam6i5MpEFBtlnxiT0eRtf pKRYwjWJmQUYdmgtE87bsgKzMIxEZ/LOLjqXxtKRQbBubPAVPulwH55Y6T/b0WYqiOBP 4w/u9g41FDQlO+ggZxgh9VrDzC2xOCRazXJ06yD0voBqeFczaKoDNJB8kNsUJKKsdflc 6YQrKcLrOg7H8bDxsGiyyBCz2IPuF1ZquBEIYqa7QBNxBMjV7/dTXmU+ZkzPRFya1PnN SQkmQ9ZB5PokA3tksUFxFCdmX+baEAV34148+pHQhpNyvw7oYAMpjeo9xRNFvDm0fH7f LCpw== X-Gm-Message-State: APuU4Z92PbCxf6iEcWPSu8caVfTEafMQXxrrvowQaxQJRCVBP00W4zWc OFeiN6JEmig+evdsSME2bGudX1YVGqeNEGFo9/yrVBB8nidXKBfRHqBQvy0ZMuaw X-Gm-Gg: AYBFou32ForbNITcOpT+FYmGaMolbYK7P6XXWH0F8e9bK7qDK18yFKk/cvjHHVCCopS 8+VX8AZjE653zaViuARk1Z094s/KqZ99K70dSlaZ+30jnbDTkyov340azboQyK/+TxqTSDNBpud KB+nwzn62z7+ckJmmRhW18l/JoTymLvBuGLXibvSd4ZK2TsUeJNah2YGaKBnr4sKlgD3FF0+J/t jXHBBQTzfsFUACYScGlaI4QswnSlT0Z4GeCQfJbScyuhXbpRGSAKR80Pk03cn4xyw6+Y3Tw+p16 o7T9UujZigswBdPckGYtF/DnHe16jBp1a1OIJabtPIWrAhsKSMq0SmZyfOQccKcCm47JcjwGLWU qYOt16NxZHLn/KaGXm9cFlwuaU8JEsiXm9EggSsm1XQw2+01ssQpLa08imuC/g7FB0BRqCqN6Tu Mo2buHAzGw2dg+z3tL2p6B89sNuUmlRRL76VgKAPyiwLFOZcgoBoeLEMArNL8hgRW771171Kp+4 e/IG0NpVVf0SSQ2Rm8LHliSUERAbQF/10CIo67VsafCNunk5iGadTsrD8qTZlFHqQKYNfLI4I4X UVGB+GhzwJs859T42encJZqqKZDMS3MubDkYDN618ze29KXfbvsB+aonNreL6TxlIW+XuwQu8uo Ih5PafLWyqjDbNdzJzHmFP1rHFdgs X-Received: by 2002:a05:690e:168d:b0:679:a2f4:386e with SMTP id 956f58d0204a3-67c67c91b3bmr2905046d50.9.1794434289796; Wed, 11 Nov 2026 13:58:09 -0800 (PST) From: Lucas Barros To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager] fix #8240: lrm: keep try count when re-queuing a waiting command Date: Wed, 11 Nov 2026 18:58:05 -0300 Message-ID: <20261111215805.8414-1-lucasbarros.kam@gmail.com> X-Mailer: git-send-email 2.50.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: X5ZH264TURJ3IREX64SONKSVRBRSW536 X-Message-ID-Hash: X5ZH264TURJ3IREX64SONKSVRBRSW536 X-MailFrom: lucasbarros.kam@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Commit 65c1fba ("lrm: avoid job starvation on huge workloads") made run_workers() prefer queued commands that were already passed over more often, so that every resource gets its turn eventually when not all queued commands can be started within one LRM round. But manage_resources() queues the current command of every resource again in each round, and queue_resource_command() replaces a queued, not yet started command with a fresh entry with 'start_tries' reset to 0. So at the start of every round all waiting commands are equal again and the sort falls back to the service ID: with more resources than the LRM can handle in one round, the lexicographically first ones are checked every round, while the others are never started and stay in their state, e.g. 'starting', for as long as the load lasts. Keep the try count of the replaced entry, so that commands that waited longer are actually started first. Signed-off-by: Lucas Barros --- The simulator based tests use max_workers 0, where every command is run directly, so they do not cover this. To check it, I drove the real manage_resources()/run_workers() of the LRM with a fake environment on a PVE 9 node: 60 'started' services, 4 workers, each resource agent call takes one second, and, like the CRM, a service gets a new uid whenever the LRM reported a result for the current one. Over 4 rounds (3 runs each): before: checked at least once: 29, 31, 33 of 60 never checked: vm:130 ... vm:160 (always the end of the sid order) after: checked at least once: 60, 60, 60 of 60 The script is ~60 lines, I can post it if that helps. Only the try count of an entry that was not started yet is carried over; a new entry still starts at 0, and the count is dropped together with the entry once its worker finished. If the replaced command is for a different state, it currently inherits the count as well, as the resource waited for its turn either way. I can restrict it to the same state if preferred. The series "some sid parsing and LRM speedup improvements" (Daniel Kral, July) does not touch queue_resource_command() or the ordering, so this should apply on top of it as well. 'make test' in src passes before and after (153 of 153 regression tests and the 9 unit tests). proxmox-perltidy reports no changes. This patch was prepared with the help of an LLM based coding assistant; I reviewed the change and ran the tests mentioned above myself. src/PVE/HA/LRM.pm | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm index 72e37e6..2105593 100644 --- a/src/PVE/HA/LRM.pm +++ b/src/PVE/HA/LRM.pm @@ -773,9 +773,12 @@ sub queue_resource_command { # with no active CRM) return if $state ne 'stopped' && $uid && defined($self->{results}->{$uid}); + my $start_tries = 0; if (my $w = $self->{workers}->{$sid}) { return if $w->{pid}; # already started - # else, delete and overwrite queue entry with new command + # else, delete and overwrite queue entry with new command, but keep the try count, so + # that the fair scheduling in run_workers() still prefers it over newly queued jobs + $start_tries = $w->{start_tries} // 0; delete $self->{workers}->{$sid}; } @@ -783,7 +786,7 @@ sub queue_resource_command { sid => $sid, uid => $uid, state => $state, - start_tries => 0, + start_tries => $start_tries, }; $self->{workers}->{$sid}->{params} = $params if $params; -- 2.47.3