From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 920CE1FF09C for ; Mon, 02 Nov 2026 10:54:58 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7F91221696; Mon, 02 Nov 2026 10:54:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linbit.com; s=google; t=1793613280; x=1794218080; darn=lists.proxmox.com; h=user-agent:content-disposition:content-type:mime-version:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=aKuGxw0QHyPwayCc2NmYx3CEj68hh6n53g6hmNsL/RM=; b=qQNcNNclgffPf3Z7gs8Mzx6yieaQhTYgbIf28Ouoq8Pg1VPQrgnNWYbBhDTnEuRKhJ dO8d15xVN9pX8XNW9pRG16aC44we+rh3zG+w58jFnD2GtY2QhmEtUDMAaZjRLXG11oOt lRvJ43FAKvhJn4pV2MLp3N6hf6ryzU43m4W6gwpWEcAUvsqYzbhUXu54uKDr4NInm0n0 x+Mes3QNYMr8o6fAyLa7jymemktWbsrp6z9neMgQNtdNdjmrXAdspU6BJWOUH14SmfRs p7SdiZYDOQI2lMcKkG+Pye9d5oD1X/1Gk9luE+KGFyITnd4hUFJt4PrJwXi77Argzg81 vgvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793613280; x=1794218080; h=user-agent:content-disposition:content-type:mime-version:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to:content-type; bh=aKuGxw0QHyPwayCc2NmYx3CEj68hh6n53g6hmNsL/RM=; b=pidQyemva6/1rZsuFd9FHquaTEm9O4455gE9+eTIQM7PHO1XzetHSwpqQ1/wNpoTJc 7R9LzNuwLaaXyTapwHuHLkZYSISSrFBs+1g2QZQpbOPoFB/aM1SmEJcLrVrmmG9DxLxW o0S+4dPmdNXKW7y6iQCR6rcJOPruvpUZzr/TJpjgOnAdRiDZBkQhHl1zFtHFxwCYteTt o+tNQpBz5YQrGshE8MtIqY2IdTlX3aXFF3YC6EAU/M4wgkgb+AOogGU4A/BatssxbvXv YjVX+H27kuxFB1y/fE03gLNuTNA/UoK/NBnyEaPT9yx1RiWm/Xaj9v+dIMRzTm7vHffi UKcw== X-Gm-Message-State: APuU4Z+UJIIftpi/tB662QhXQI5m4YNzZnDjX+RIvJuIOqN9yUksXth9 hSR+2IEh5npK8Umst37QzhYtd5/mrDwpx6mpM3SvuxFO+Cj4QIeNMkR2Ds52i5CTf7WuyQfi9q4 aXuKS2NTGWg== X-Gm-Gg: AYBFou1hopge3QgzjmG44iNlEiAIabKGvSp9KB7zGEhLWlHi+tcvAYbnHTaaJ0mppGI fvOnIr7ocHCNz2xMnmw+5PAwz0qjHlhx7Nlc1pt3jwTdCZntjEkKANu34sqjEBR3f5IaYSrrcHT S0cxYUWdJnJRDDOY8+f48R2qmN+MRri/NH/1pzRYXeZaz5ivFmyoMeRYh9cHrysbyUV30+nsYKY RUHzMzdPFsY4sxNgEJRxZD92tdEVfn9X+RNU/MATFN3vLaVlwuJowzGe7npJKzlZ1blFl1eMwzp +nHDfyIxPX+7UGVOgRJhrAixCbNGem4//eotgWq1LUb7GYmNEsH9rIlQ7svd+B9XCc+Riog6rQO 8hD6KtZLtAzfgq8d46+yKc948/AK+Dr0chddrIRXd5UnPC6HSyP4XP74FIbW3mkMVuxZDdRxrD1 mTF1/v0moVdBSX0LzpDKZRj4qWRO742wbreot3gMq1KwVTtzuhHw+A2Pdy742P6Ig3ogtunJPNu cyyZwUnVW2r0F1a2qKVWjippUp+EZvd2tbwxKDk1lcdVHQug7yx3F/VXAbfxYUv3XHYVw3GlEzB CzPrz9fjwDX1e8CDzkti7e/ZxPPxdpb4sEYRGjrFW4e5xG6BarqlTjKiV0ktEyphZqzBQ9oLUiX h+JGc X-Received: by 2002:a05:6512:124c:b0:5bf:d7e4:3c4e with SMTP id 2adb3069b0e04-5bfd95991c8mr3660410e87.51.1793613279741; Mon, 02 Nov 2026 01:54:39 -0800 (PST) Date: Mon, 2 Nov 2026 10:54:38 +0100 From: Roland Kammerer To: pve-devel@lists.proxmox.com Subject: [RFC] storage plugin hook for node preference in the load rebalancer Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/2.3.3 (2026-06-12) X-SPAM-LEVEL: Spam detection results: 0 AWL 0.200 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 6NE4VDSWWZVS7JBP4K6273T7DRM6PWFP X-Message-ID-Hash: 6NE4VDSWWZVS7JBP4K6273T7DRM6PWFP X-MailFrom: roland.kammerer@linbit.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi PVE devs, Daniel, thanks for the interesting talk at Proxtalks and our short discussion in the Q&A. The tl;dr version is: I think it would be great if the storage (plugin) could have a say in where a VM is placed/rebalanced. LINSTOR/DRBD would definitely benefit from that, there is some prior art on other platforms like stork (Portworx) on k8s and I hope this could benefit other storage providers as well. The following then contains some RFC in parts generated with an LLM, but first let me state the purpose of the RFC as of now and the LINSTOR use case: Don't be too picky about the proposed API, that is a first draft with a simple list, it was the first thing that came to my mind. We could have weights per node or whatever, right now I'm interested if "asking the storage for placement" is something you would be willing to consider in general, the actual API is for now "technical details". The LINSTOR/DRBD case: Imagine a hyperconverged cluster where the PVE nodes are the LINSTOR nodes and the user configured LINSTOR/DRBD replica count of 3. The actual storage then lands on 3 nodes. At creation time the plugin already tries to create local storage on the node the VM is created. If the VM is then later moved it could end on a node that has a local copy or not. In every case the actual storage can be accessed. If we don't have local storage we need to read/write over the network, if it is a node with local storage we can serve reads locally, a huge benefit. If the storage could weigh in on placement decisions that would be great. Now the LLM assisted part, take it with a grain of salt, not every use case might be valid and there might be ones that got overlooked. You certainly know your ecosystem better than me, personally I only know stork (Portworx) on k8s where the "ask the storage what it thinks" exists for sure: With the dynamic scheduler + load rebalancer series [0] the HA manager will pick (resource, target node) pairs on its own and queue live migrations. Candidates are filtered by HA rules and scored on CPU/memory imbalance only. The storage a guest lives on has no say in where the guest ends up, although for several storage types the target node matters: - ZFS with storage replication: live migration to a non-target node is refused by qemu-server (or does a full copy with --force), to a target node it is incremental. - DRBD/LINSTOR: full-copy replicas on a few nodes, every other node goes over the network (same model as Portworx on Kubernetes, where stork steers pods to nodes holding a replica). - Ceph/Vitastor with localized reads in stretched setups: moving the VM to the other site turns every read into a cross-site read. - Shared SAN (iSCSI/FC + LVM): per-node path health is visible to the storage, not to the HA manager. Proposal: a small, optional hook in the storage plugin API. # PVE::Storage::Plugin, default implementation sub rank_nodes { my ($class, $scfg, $storeid, $volname, $nodes) = @_; return $nodes; # no preference } The caller passes the candidate nodes (e.g. [D, B, C]), the plugin returns them ordered by preference (e.g. [B, D, C]). Nodes left out are unsuitable, so the hook covers both hard exclusion and soft preference. A wrapper in PVE::Storage aggregates over all volumes of a guest. The rebalancer can drop excluded candidates before scoring and feed the rank into the planned "migration cost" criterion. The same hook would help HA recovery and rebalance-on-start, where a replicated ZFS guest can today be recovered onto a node without a copy of its disk. This mirrors CSI topology / volume node affinity in Kubernetes, where the scheduler asks the storage layer before placing a workload. Open questions: - Ordered list, or a numeric cost per node for mixing with other TOPSIS criteria? - Aggregation across volumes of one guest in PVE::Storage or in the HA manager? We would implement this for the LINSTOR plugin and are happy to send patches for the default implementation and the wrapper if there is interest. Best regards, Roland [0] https://lore.proxmox.com/pve-devel/20260330144101.668747-1-d.kral@proxmox.com/