[RFC] storage plugin hook for node preference in the load rebalancer

[RFC] storage plugin hook for node preference in the load rebalancer

[Roland Kammerer]

Hi PVE devs,

Daniel, thanks for the interesting talk at Proxtalks and our short
discussion in the Q&A.

The tl;dr version is: I think it would be great if the storage (plugin)
could have a say in where a VM is placed/rebalanced. LINSTOR/DRBD would
definitely benefit from that, there is some prior art on other platforms
like stork (Portworx) on k8s and I hope this could benefit other storage
providers as well.

The following then contains some RFC in parts generated with an LLM, but
first let me state the purpose of the RFC as of now and the LINSTOR use
case:

Don't be too picky about the proposed API, that is a first draft with a
simple list, it was the first thing that came to my mind. We could have
weights per node or whatever, right now I'm interested if "asking the
storage for placement" is something you would be willing to consider in
general, the actual API is for now "technical details".

The LINSTOR/DRBD case: Imagine a hyperconverged cluster where the PVE
nodes are the LINSTOR nodes and the user configured LINSTOR/DRBD replica
count of 3. The actual storage then lands on 3 nodes. At creation time
the plugin already tries to create local storage on the node the VM is
created. If the VM is then later moved it could end on a node that has a
local copy or not. In every case the actual storage can be accessed. If
we don't have local storage we need to read/write over the network, if
it is a node with local storage we can serve reads locally, a huge
benefit. If the storage could weigh in on placement decisions that would
be great.

Now the LLM assisted part, take it with a grain of salt, not every use
case might be valid and there might be ones that got overlooked. You
certainly know your ecosystem better than me, personally I only know
stork (Portworx) on k8s where the "ask the storage what it thinks"
exists for sure:

With the dynamic scheduler + load rebalancer series [0] the HA manager
will pick (resource, target node) pairs on its own and queue live
migrations. Candidates are filtered by HA rules and scored on CPU/memory
imbalance only. The storage a guest lives on has no say in where the
guest ends up, although for several storage types the target node
matters:

- ZFS with storage replication: live migration to a non-target node is
  refused by qemu-server (or does a full copy with --force), to a target
  node it is incremental.
- DRBD/LINSTOR: full-copy replicas on a few nodes, every other node goes
  over the network (same model as Portworx on Kubernetes, where stork
  steers pods to nodes holding a replica).
- Ceph/Vitastor with localized reads in stretched setups: moving the VM
  to the other site turns every read into a cross-site read.
- Shared SAN (iSCSI/FC + LVM): per-node path health is visible to the
  storage, not to the HA manager.

Proposal: a small, optional hook in the storage plugin API.

    # PVE::Storage::Plugin, default implementation
    sub rank_nodes {
        my ($class, $scfg, $storeid, $volname, $nodes) = @_;
        return $nodes; # no preference
    }

The caller passes the candidate nodes (e.g. [D, B, C]), the plugin
returns them ordered by preference (e.g. [B, D, C]). Nodes left out are
unsuitable, so the hook covers both hard exclusion and soft preference.
A wrapper in PVE::Storage aggregates over all volumes of a guest. The
rebalancer can drop excluded candidates before scoring and feed the rank
into the planned "migration cost" criterion. The same hook would help HA
recovery and rebalance-on-start, where a replicated ZFS guest can today
be recovered onto a node without a copy of its disk.

This mirrors CSI topology / volume node affinity in Kubernetes, where
the scheduler asks the storage layer before placing a workload.

Open questions:

- Ordered list, or a numeric cost per node for mixing with other TOPSIS
  criteria?
- Aggregation across volumes of one guest in PVE::Storage or in the HA
  manager?

We would implement this for the LINSTOR plugin and are happy to send
patches for the default implementation and the wrapper if there is
interest.

Best regards,
Roland

[0] https://lore.proxmox.com/pve-devel/20260330144101.668747-1-d.kral@proxmox.com/