From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 6AF521FF0AD for ; Sun, 18 Oct 2026 14:46:42 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DE76B21523; Sun, 18 Oct 2026 14:46:30 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XVOUGamKZyvjb2FfLghisSHKgXl5KJkpKazkYPyzutUfQvudK56UYo1M1e6t2f5p2Pct8STiQ224E3sj2FiHX1fpRL6/3UuNivJ1k9YNpEFv/q2rQREmfDVhsPGZ9USGwO293ERbs4mendc41rua2FYy5yKFJn+PxTK91aznyWtYqdVwMVNsdG8hKUW0mezdTDWlbcnkQa0DmGm5XaWJQhfHWUsNBt2s0eDLOSe1TJLSP+nbqy6sGJ3fVT2dfFtN1uNuZZ4vAzv1eUCjNVRxX32MM6UI2AxhoZc+pKBL0BlWCj9W5LS4j5IIR2dK0bgT2NTaDCYflExujPNDmurYkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tFMUQ+99bFVq+f0STns5iFHHf+ERo2z4rQ2nrr69QXY=; b=yaiMXnS7GdQnKRI8y9Em7wykZ6FkWrbz8k0k9M8Y78MC+ukrnlKdq3b7JkqBvb0G7gyylIqcbX4AxkbLVzsy/pXYx6uArKfdyhaVHn1LxYBXEeLdGL/RW2JA1jv2aFpPjDkXka6XFzYfRKAffKHjDVO0bsD5b77+VlQ4+/oFYwsz7LgguYirJZxM8WRn+62jxeYyD3d26o5dcqjOKSz1sdpYPWo0I6S4fhAAOPSKD6mrV0dJ/PniJOJKDLvEe5pEzl5RMdMwP6tdGHqkzWsBRwuoF0b7IA7JkjBu5ICf20i3tGnEZ5uJUaH6Md8a5T/xQUqXpqt1hqMpkGnzC9282A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tFMUQ+99bFVq+f0STns5iFHHf+ERo2z4rQ2nrr69QXY=; b=A06LvALWsfdGHPhFBGw/JpXvLVgR9sgnOMGy0sNRYH2elYwcSbvzgtb553m1ay8UEbedCNVi1tQwGhYOUaahkbk/EPGtnQUsiwyZioCwM7gHb9Iay0nKzjlbegvyt4XPQIQFYEs2b1nkmxDYPvsO5hX+Tj4liI1TAFr7zTCz7B9MuileHoyuLKZLJKX4Je/M67wVtKa6XCBYGWrulCoZsixrUygLjftHwRGPuFMmrcgWiJtdXJX2EirFT2IGcoUEAB+sL3v0AKluu6x4caY9LnsJfAkKXOqmX/7JUuHVH9WA4ahJyY/Ng/rqQZ4YdyjUznRRt/OekezyD4aSfCXTFg== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager v2 03/14] test: add auto rebalance test cases for failing rebalance motions Date: Sun, 18 Oct 2026 14:45:47 +0200 Message-ID: <20261018124559.24305-4-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261018124559.24305-1-Michael@RyomHerold.dk> References: <20261018124559.24305-1-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MM0P280CA0046.SWEP280.PROD.OUTLOOK.COM (2603:10a6:190:b::20) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|PA4PR10MB5706:EE_ X-MS-Office365-Filtering-Correlation-Id: d7e25e11-ddb3-417b-4787-08df2d15cbfb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|23010399003|6133799003|22082099003|10067099003|18002099003|260925021311599003|261009223027099003|260925022911599003|260925021911599003|56012099006; X-Microsoft-Antispam-Message-Info: jx4dp4eoekgoPpyuv1ZT8s5U1iV4fwz1ossz6hKOHCfqtIc3bT/LcEv7TmEDpsfSCxhYR75Jd0s+vXP2G/xFSVGNM+TwVotDe2amUd7Alehq4GF+aLVzLPbKTcLcvgqDzoNXCJzKkJC4lLjIVhKehAOKIreU4+1sawcHDHmJIXFjRp0nu7pHYrwlKpKTWMo2RrKM1cAtJRLkibQtJPc39e5jylbeKA6jUamWWv0kqL2aN+u1XS3PrzehrkOAQvud3bRC3wgzaErvEFYlVz8TMPdelAZeV8ZMnS8hzRoyEGpGvip6sDElo9jzTq+gP/5JdYcKYiM+ij+sKYd3nOA1ZgR7zVQw/EKWHPPJu5yK0m0LMLDMSZhBza0Y9ccWxyVcDrXfyznb1jBppx/gOXliSw16041zWvrFSiFOKT+ErcTlU8kV2UxUA3+8WM2UU2PmuFhQIcJa6aztyQQcYcv2SNd94cmXU+GtlbSqyXqpalLlUZpo3DXZlMnxNweK87xMdNC7T54fZz+4eFk8qdX57CJf/bbsk8fArZPjZroYBQoiFY8JvQukqTnkRWnDA55vKV9PRbeWSUfALsuSmpbg9W0Jam0ywgjlz56y3rflUH0o2Dhr+1CBtrNAq2GicUArHODeE5d/D2x9YnooaIOFeWDIIE6brOx1Ae1adFS2/74= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(23010399003)(6133799003)(22082099003)(10067099003)(18002099003)(260925021311599003)(261009223027099003)(260925022911599003)(260925021911599003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?c35OUPl4610lto4E2vOhRRTtTUzpObei2PH3MmpG/5epj6+f/pydCyVAAzRx?= =?us-ascii?Q?NtewhBj50ALPR1Qq/uGE0SgkkrnJ2DwoOJGbCCbNxFVJPMV67pvfSi5yhGzM?= =?us-ascii?Q?5xb2UZkx9Iuauc56Z+YxYvN/LaNJTxO3rMwjGEZWFapPgJFjmMNchmLPJXDC?= =?us-ascii?Q?/qapgzmJ2l87BRybD7F9uDUOoyLOweDslGA1crMvINu0gGvFhASOQYyO7JUu?= =?us-ascii?Q?kffYTTA9/j/4tCa47kMSY90KZtIYplCuqbtpYvZewW1Fv1b+zLqycXQw4Uc/?= =?us-ascii?Q?c01FX34XKN5vad0XUYg5bM37yMI3TK+H9KHwSllss1WWPzkHg1UNkw+wFSAl?= =?us-ascii?Q?wPl9cENYES0nEonzC40tnl42QZNWCamsPUYV3v2+b6GSyQugKmXxfRF0reku?= =?us-ascii?Q?ZIw1wAHzyjTSnmXLhm7SwUhEnrCznN32fR93cL9UcCVEtOqStT9cMJ1+D6aH?= =?us-ascii?Q?P6JPhtV2kZqqX4B5BT5nPLaNTxfkK4C1bF+F985N1brIcCsVgi6rSFV40+o3?= =?us-ascii?Q?3M5YkMpEfx0AUGYLirzA8feafenDTfN6YT/TBb6OBvoduqnW0qhMdMEM3fEF?= =?us-ascii?Q?WTCCkqjLv8+xq5q5rOK2xIs/0jQIr11NNR5HkTvK5y8G54cj6O2gkh0+/+Lg?= =?us-ascii?Q?La7MwkeFw5ROiFmABdNq+lIS1syYeuEuLaqX2iM4niWwl9ci3hy0roGMH78q?= =?us-ascii?Q?n5K4Q/MWP+FHHbKlfAppVTkO/wn7tL9QJlqH45Y7k2U9dqhZX0cTytrozkJ4?= =?us-ascii?Q?E7BpbTtuuPYCKeK0WR0sEvIIs//FrESWucQJEuMIe2gOGzWJfCZby2tEqn2I?= =?us-ascii?Q?Kn4lDOnPCU6qFw4JkMmjcczKJCbWJdZGhtVVaiWaRGnS7PjuzkR/RC7u0/r6?= =?us-ascii?Q?Cf/gg3ZcWu5XISUAocdavGRI5gYJoPXXDtYONYmZdP+RupbVLvS9uhpfiUSh?= =?us-ascii?Q?vILJY2Qr1e0g8BKNi0CG7kLa/lg7dWCK2F8LuAcX3ZCDiJTubK6SbN4imoZb?= =?us-ascii?Q?wvl3YFnmOSVoPBzC2p+uFp1z7yOBDaGQzfDq6xQ/WK/KomOvo9KgSUIrMXHq?= =?us-ascii?Q?ws/xI0qm8CCjZKQ/62tlCXKXqJS+BrHz3DR/GOkJ4ioKpHv/wvLG3pgmGF0i?= =?us-ascii?Q?6putvpf9Pm8I0sb2leO8aqu2LQ5LfCdAtdv98hKI8waK7LoNg3oPoQ9Dgmd9?= =?us-ascii?Q?Ui85MD9z4gKJsPyIKphlVSBqNFeSeJ/p7/mu5shORI1fTj9LIg05W5sG/kp4?= =?us-ascii?Q?aO/j1KRhjJipcOO6AcMRII+p2zhYEB+HSVr7xMg4z9zKSTdSth3kZmrpqN5d?= =?us-ascii?Q?uF8Lj9yZckqr3X4BbgQxrxBnkC9w7gW0Lo0KPYkjy82ea8Oopv9L/tGWnFDI?= =?us-ascii?Q?xbhZ1p+X+QUub48CwFEF22mguHXCT3mI5T7VwIFMBbQ8kgFUq0+/s8ZtTtT7?= =?us-ascii?Q?n7xYTv8/1dBoyQkbJ5ITauJJcqiQoHHwneEtdUVHSuhQVqzMRXd2bdOVTKax?= =?us-ascii?Q?My+BeQbnucIvfAb/Y9NVV0tVhp5GD91b8un48dFNxbkDRxNHzkXu0oWP/bmE?= =?us-ascii?Q?K8s+ciAozvHUAML8bkRL1Cgf4BzrxjlTGP2UKMlAjuHz7wjNzD+rsUIR05YQ?= =?us-ascii?Q?qLpCX16kjRgb/tFW09cFr3B1JSr35SNpKKnaO6A0FPClBOSObPDm1fIbqDqI?= =?us-ascii?Q?UeMJ15Uzj2NSQB4gKwmxGlSLVAvU48vrMbRIXsOa4DW36kqluhRyxO2y902u?= =?us-ascii?Q?s67+dr2gOA=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: d7e25e11-ddb3-417b-4787-08df2d15cbfb X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2026 12:46:15.0884 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 777oJXmv0DvATd35td3HWLmO5F2Pb6mFgn88fNdUnqUI59BVjOdKEBOkDN6RE37BUyVMdwLWsKoBWBynqvMnJA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB5706 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.004 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy GB_MSAUTHSOURCE_EUROPE 0.01 Mail authenticated via Microsoft Europe (EURP) datacenter RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: H2O7ZDHV3CIKRPQ4SN3UEW6ADERHN4J3 X-Message-ID-Hash: H2O7ZDHV3CIKRPQ4SN3UEW6ADERHN4J3 X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The automatic load balancer does not record the outcome of the motions it issues. If a motion fails, the resource is put back to 'started' on its source node and the very same motion is selected again as soon as the imbalance has exceeded the threshold for the hold duration again (see bug #8059). Add two test cases which show this behavior: - test-crs-dynamic-auto-rebalance5: a failing relocation of fa:109 is retried every 80 seconds, even though migrating any of the other HA resources would improve the imbalance just as well. - test-crs-dynamic-auto-rebalance6: the failing relocation of fa:109 is the only possible rebalance motion and is retried every 80 seconds as well. The expected logs record the current behavior and are adapted by the following patch. Signed-off-by: Michael Ryom --- .../test-crs-dynamic-auto-rebalance5/README | 9 ++ .../test-crs-dynamic-auto-rebalance5/cmdlist | 3 + .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 6 ++ .../hardware_status | 5 + .../log.expect | 88 +++++++++++++++ .../manager_status | 1 + .../service_config | 6 ++ .../static_service_stats | 6 ++ .../test-crs-dynamic-auto-rebalance6/README | 10 ++ .../test-crs-dynamic-auto-rebalance6/cmdlist | 17 +++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 4 + .../hardware_status | 4 + .../log.expect | 100 ++++++++++++++++++ .../manager_status | 1 + .../service_config | 4 + .../static_service_stats | 4 + 18 files changed, 280 insertions(+) create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/static_service_stats diff --git a/src/test/test-crs-dynamic-auto-rebalance5/README b/src/test/test-crs-dynamic-auto-rebalance5/README new file mode 100644 index 0000000..018e604 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/README @@ -0,0 +1,9 @@ +Test that the auto rebalance system does not retry a failing rebalance motion +over and over again. + +The HA resource fa:109 fails its first nine migrate/relocate attempts. The +auto rebalancer selects fa:109 to be relocated, which fails, and fa:109 is put +back to 'started' on node1. As long as the imbalance stays above the threshold, +the auto rebalancer should neither select the very same failed motion again +nor keep alternating between failing motions of fa:109 to the other nodes, but +try the next best rebalance motion, i.e. migrate one of the other HA resources. diff --git a/src/test/test-crs-dynamic-auto-rebalance5/cmdlist b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist new file mode 100644 index 0000000..13f90cd --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist @@ -0,0 +1,3 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats new file mode 100644 index 0000000..921df6a --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats @@ -0,0 +1,6 @@ +{ + "fa:109": { "cpu": 1.0, "mem": 4294967296 }, + "vm:102": { "cpu": 1.0, "mem": 4294967296 }, + "vm:103": { "cpu": 1.0, "mem": 4294967296 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/hardware_status b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status new file mode 100644 index 0000000..ce8cf0e --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/log.expect b/src/test/test-crs-dynamic-auto-rebalance5/log.expect new file mode 100644 index 0000000..a5dad5a --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/log.expect @@ -0,0 +1,88 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'fa:109' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node1' +info 20 node1/crm: adding new service 'vm:104' on node 'node1' +info 20 node1/crm: service 'fa:109': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node1) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service fa:109 +info 21 node1/lrm: service status fa:109 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 21 node1/lrm: starting service vm:103 +info 21 node1/lrm: service status vm:103 started +info 21 node1/lrm: starting service vm:104 +info 21 node1/lrm: service status vm:104 started +info 22 node2/crm: status change wait_for_quorum => slave +info 24 node3/crm: status change wait_for_quorum => slave +info 80 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 80 node1/crm: got crm command: relocate fa:109 node2 +info 80 node1/crm: relocate service 'fa:109' to node 'node2' +info 80 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 81 node1/lrm: service fa:109 not moved (migration error) +info 83 node2/lrm: got lock 'ha_agent_node2_lock' +info 83 node2/lrm: status change wait_for_agent_lock => active +err 100 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 160 node1/crm: got crm command: relocate fa:109 node2 +info 160 node1/crm: relocate service 'fa:109' to node 'node2' +info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 161 node1/lrm: service fa:109 not moved (migration error) +err 180 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 240 node1/crm: got crm command: relocate fa:109 node2 +info 240 node1/crm: relocate service 'fa:109' to node 'node2' +info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 241 node1/lrm: service fa:109 not moved (migration error) +err 260 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 320 node1/crm: got crm command: relocate fa:109 node2 +info 320 node1/crm: relocate service 'fa:109' to node 'node2' +info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 321 node1/lrm: service fa:109 not moved (migration error) +err 340 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 400 node1/crm: got crm command: relocate fa:109 node2 +info 400 node1/crm: relocate service 'fa:109' to node 'node2' +info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 401 node1/lrm: service fa:109 not moved (migration error) +err 420 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 480 node1/crm: got crm command: relocate fa:109 node2 +info 480 node1/crm: relocate service 'fa:109' to node 'node2' +info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 481 node1/lrm: service fa:109 not moved (migration error) +err 500 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 560 node1/crm: got crm command: relocate fa:109 node2 +info 560 node1/crm: relocate service 'fa:109' to node 'node2' +info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 561 node1/lrm: service fa:109 not moved (migration error) +err 580 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 620 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance5/manager_status b/src/test/test-crs-dynamic-auto-rebalance5/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/service_config b/src/test/test-crs-dynamic-auto-rebalance5/service_config new file mode 100644 index 0000000..3685b14 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/service_config @@ -0,0 +1,6 @@ +{ + "fa:109": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node1", "state": "started" }, + "vm:104": { "node": "node1", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats new file mode 100644 index 0000000..6dab3d1 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats @@ -0,0 +1,6 @@ +{ + "fa:109": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/README b/src/test/test-crs-dynamic-auto-rebalance6/README new file mode 100644 index 0000000..76e9411 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/README @@ -0,0 +1,10 @@ +Test that the auto rebalance system retries a rebalance motion, which failed +before, less and less often if it keeps failing and there is no other +rebalance motion to fall back to. + +The HA resource vm:100 is excluded from auto rebalancing and only acts as a +node base load on node1. The only movable HA resource fa:109 fails its first +nine migrate/relocate attempts, so the auto rebalancer can only ever select +the failing relocation of fa:109 to node2. The time between two attempts should +grow with each failure instead of retrying it every time the imbalance has +exceeded the threshold for the hold duration. diff --git a/src/test/test-crs-dynamic-auto-rebalance6/cmdlist b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist new file mode 100644 index 0000000..c8c644e --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist @@ -0,0 +1,17 @@ +[ + [ "power node1 on", "power node2 on" ], + [], + [], + [], + [], + [], + [], + [], + [], + [], + [], + [], + [], + [], + [] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats new file mode 100644 index 0000000..77ef5c2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats @@ -0,0 +1,4 @@ +{ + "fa:109": { "cpu": 1.0, "mem": 4294967296 }, + "vm:100": { "cpu": 4.0, "mem": 4294967296 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/hardware_status b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status new file mode 100644 index 0000000..864adb3 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status @@ -0,0 +1,4 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/log.expect b/src/test/test-crs-dynamic-auto-rebalance6/log.expect new file mode 100644 index 0000000..9758af5 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/log.expect @@ -0,0 +1,100 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'fa:109' on node 'node1' +info 20 node1/crm: adding new service 'vm:100' on node 'node1' +info 20 node1/crm: service 'fa:109': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service fa:109 +info 21 node1/lrm: service status fa:109 started +info 21 node1/lrm: starting service vm:100 +info 21 node1/lrm: service status vm:100 started +info 22 node2/crm: status change wait_for_quorum => slave +info 80 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 80 node1/crm: got crm command: relocate fa:109 node2 +info 80 node1/crm: relocate service 'fa:109' to node 'node2' +info 80 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 81 node1/lrm: service fa:109 not moved (migration error) +info 83 node2/lrm: got lock 'ha_agent_node2_lock' +info 83 node2/lrm: status change wait_for_agent_lock => active +err 100 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 160 node1/crm: got crm command: relocate fa:109 node2 +info 160 node1/crm: relocate service 'fa:109' to node 'node2' +info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 161 node1/lrm: service fa:109 not moved (migration error) +err 180 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 240 node1/crm: got crm command: relocate fa:109 node2 +info 240 node1/crm: relocate service 'fa:109' to node 'node2' +info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 241 node1/lrm: service fa:109 not moved (migration error) +err 260 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 320 node1/crm: got crm command: relocate fa:109 node2 +info 320 node1/crm: relocate service 'fa:109' to node 'node2' +info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 321 node1/lrm: service fa:109 not moved (migration error) +err 340 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 400 node1/crm: got crm command: relocate fa:109 node2 +info 400 node1/crm: relocate service 'fa:109' to node 'node2' +info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 401 node1/lrm: service fa:109 not moved (migration error) +err 420 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 480 node1/crm: got crm command: relocate fa:109 node2 +info 480 node1/crm: relocate service 'fa:109' to node 'node2' +info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 481 node1/lrm: service fa:109 not moved (migration error) +err 500 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 560 node1/crm: got crm command: relocate fa:109 node2 +info 560 node1/crm: relocate service 'fa:109' to node 'node2' +info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 561 node1/lrm: service fa:109 not moved (migration error) +err 580 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 640 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 640 node1/crm: got crm command: relocate fa:109 node2 +info 640 node1/crm: relocate service 'fa:109' to node 'node2' +info 640 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 641 node1/lrm: service fa:109 not moved (migration error) +err 660 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 660 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 720 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 720 node1/crm: got crm command: relocate fa:109 node2 +info 720 node1/crm: relocate service 'fa:109' to node 'node2' +info 720 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 721 node1/lrm: service fa:109 not moved (migration error) +err 740 node1/crm: service 'fa:109' - migration failed (exit code 1) +info 740 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1) +info 800 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%) +info 800 node1/crm: got crm command: relocate fa:109 node2 +info 800 node1/crm: relocate service 'fa:109' to node 'node2' +info 800 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2) +info 801 node1/lrm: service fa:109 - start relocate to node 'node2' +info 801 node1/lrm: stopping service fa:109 (relocate) +info 801 node1/lrm: service status fa:109 stopped +info 801 node1/lrm: service fa:109 - end relocate to node 'node2' +info 820 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node2) +info 823 node2/lrm: starting service fa:109 +info 823 node2/lrm: service status fa:109 started +info 1520 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance6/manager_status b/src/test/test-crs-dynamic-auto-rebalance6/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/service_config b/src/test/test-crs-dynamic-auto-rebalance6/service_config new file mode 100644 index 0000000..ccffeab --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/service_config @@ -0,0 +1,4 @@ +{ + "fa:109": { "node": "node1", "state": "started" }, + "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats new file mode 100644 index 0000000..7d2c248 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats @@ -0,0 +1,4 @@ +{ + "fa:109": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:100": { "maxcpu": 4.0, "maxmem": 8589934592 } +} -- 2.47.3