Re: [RFC PATCH-SERIES many 00/36] dynamic scheduler + load rebalancer

["Daniel Kral" <d.kral@proxmox.com>]

On Thu Mar 19, 2026 at 10:12 AM CET, Daniel Kral wrote:
> On Wed Mar 18, 2026 at 5:54 PM CET, Thomas Lamprecht wrote:
>> ScoredMigration's Ord only compares imbalance, so two migrations with
>> the same imbalance but different source/target count as Equal, which
>> makes the BinaryHeap output order unpredictable. Maybe use the Migration
>> field, which is already Ord itself, to break any ties here as a secondary
>> key.
>
> Thanks! I forgot to add this as a FIXME there for the RFC series.
> I had a
>
>     impl Ord for ScoredMigration {
>         fn cmp(&self, other: &Self) -> Ordering {
>             self.imbalance
>                 .total_cmp(&other.imbalance)
>                 .reverse()
>                 .then(self.migration.cmp(&other.migration))
>         }
>     }
>
> before, but while testing it seemed to not sort as expected. I haven't
> looked into this yet, though I guess that different calculations might
> end up in different exponents, which totalOrder does define as unequal
> [1].
>
> I'll briefly test this again, but sorting here in some reasonable way is
> still better than letting the order of the input data decide.
>
> [1] https://en.wikipedia.org/wiki/IEEE_754#Total-ordering_predicate

Indeed there was a slight issue about the f64 imbalance not being the
same even if it "seemed" like it is.

I ran into the problem with a pve-ha-manager test case yesterday, where
a test case was flakey and _sometimes_ reordered the seemingly same
imbalance fp number wrong:


    { sid: "vm:102", source_node: "node1", target_node: "node3", imbalance: 0.723174948891693 }
    { sid: "vm:102", source_node: "node1", target_node: "node2", imbalance: 0.723174948891693 }

Perl seems to truncate the 16th decimal place here and with some closer
inspection with some tracing, the actual values were:

    ScoredMigration { sid: "vm:102", source_node: "node1", target_node: "node3", imbalance: 0.723174948891693 }
    ScoredMigration { sid: "vm:102", source_node: "node1", target_node: "node2", imbalance: 0.7231749488916931 }

We and I make sure that any interaction with `Usage` in the HA Manager
is deterministic in the sense that we go through the services in the
same order every time, e.g., by sorting the keys. But I guess that
hashbrown's HashMap Iter is not deterministic as the struct itself takes
a RandomState and so we couldn't rely on the fp operation orders here.

I tried using BTreeMap and BTreeSet to make the iterator orders
deterministic, but this increased the runtime by roughly 10x, which was
unacceptable for larger cluster sizes. It also doesn't fix the problem
directly, because these cases of approximations errors caused by
changing the fp operations order could even happen with deterministic
calculations.

For the v2, I'll use the easy route and just truncated these numbers
before ordering these as the 16th decimal place is not a significant
digit anymore and shouldn't be relied on for ordering.