Re: [RFC PATCH qemu-server] fix #7282: allow (NUMA aware) vCPU pinning

["Elias Huhsovitz" <e.huhsovitz@proxmox.com>]

On Wed Sep 9, 2026 at 3:52 PM CEST, Elias Huhsovitz wrote:
> On Thu Mar 12, 2026 at 11:29 AM CET, Dominik Csapak wrote:
>> I did try to do a few benchmarks methodically but the results are
>> not really what was expected.
>>
>> Either my methodology is wrong (wrong benchmarks, etc.) or
>> the platform where I tested is actually not suited for this kind of testing.
>>
>> Still writing it down, so maybe someone else can give feedback ;)
>
> I ran some of my own benchmarks. Comments inline.


Upon reading the code in more detail and learning more about NUMA in general i have revised my benchmarking methology.

Host System
-------------
OS: Proxmox VE 9.2.18 x86_64
Kernel: Linux 7.0.14-16-pve
CPU: AMD EPYC 7351P 16-Core Processor
Memory: 64 GB DDR4 memory with 2667MT/s (4x16GB sticks)

numactl -H:
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 16 17 18 19
node 0 size: 15891 MB
node 0 free: 14693 MB
node 1 cpus: 4 5 6 7 20 21 22 23
node 1 size: 16124 MB
node 1 free: 15174 MB
node 2 cpus: 8 9 10 11 24 25 26 27
node 2 size: 16124 MB
node 2 free: 14838 MB
node 3 cpus: 12 13 14 15 28 29 30 31
node 3 size: 16123 MB
node 3 free: 14978 MB
node distances:
node     0    1    2    3
   0:   10   16   16   16
   1:   16   10   16   16
   2:   16   16   10   16
   3:   16   16   16   10

Guest System
-----------
OS: Fedora Linux 44 (Server Edition) x86_64
Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-11.0)
Kernel: Linux 7.1.13-200.fc44.x86_64
Memory: 8GB

Benchmarks Tools
----------------
* STREAM [1]

(Tool seems to be widely used, see [4,5,6])

compiled via:
gcc -O3 -fopenmp -march=native -mcmodel=medium -DSTREAM_ARRAY_SIZE=160000000 -DNTIMES=40 stream.c -o stream

executed via (stream):
OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores stream

(The OMP_NUM_THREADS could theoretically be omitted, since per default
OpenMP uses all available threads).

* sc-membench [2]

compiled via:
make numa

executed via (sc_membench):
OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores membench-numa -s 262144 -r 3 -t 30

* sysbench [3]

executed via (sysbench_mem_bw):
sysbench memory --threads=8 --time=30 --memory-block-size=512M --memory-total-size=400G --memory-access-mode=seq run
executed via (sysbench_mem_bw_local):
sysbench memory --threads=8 --time=30 --memory-block-size=512M --memory-total-size=400G --memory-access-mode=seq --memory-scope=localrun

Configurations
--------------
* 0_topology_only: guest NUMA defined, no host binding
* 1_host_numa_bound: hostnodes=N,policy=bind, no vCPU pinning
* 2_vcpu_pinned_numa: same 1_host_numa_bound + pinning=numa
* 3_vcpu_pinned_1to1: same as 1_host_numa_bound + pinning=one-to-one

Results
=======

Metric                              0_topology_only    1_host_numa_bound    2_vcpu_pinned_numa    3_vcpu_pinned_1to1
--------------------------------------------------------------------------------------------------------------------
sc_membench: copy MB/s              8388.38            8922.02              9078.42               9015.42
sc_membench: largest size KB        262144             262144               262144                262144
sc_membench: latency ns             118.26             167.54               112.56                113.38
sc_membench: latency sd ns          0.36               0.35                 0.04                  0.06
sc_membench: read MB/s              30015.90           45060.94             60461.67              60265.49
sc_membench: write MB/s             12776.54           30013.02             30904.18              30731.53
stream: Add MB/s                    34690.2            45003.3              53325.9               52752.3
stream: Copy MB/s                   43383.2            63388.4              66914.0               67128.2
stream: Scale MB/s                  30353.7            40952.8              47707.9               47004.5
stream: Triad MB/s                  34527.1            41649.4              53401.6               52790.7
sysbench_mem_bw: MiB/sec            14898.10           15583.45             10916.81              19167.89
sysbench_mem_bw: lat 95th ms        530.08             484.44               746.32                325.98
sysbench_mem_bw: lat avg ms         253.90             238.65               344.17                206.83
sysbench_mem_bw_local: MiB/sec      11173.40           20742.30             21149.99              22891.73
sysbench_mem_bw_local: lat 95th ms  308.84             227.40               189.93                164.45
sysbench_mem_bw_local: lat avg ms   174.89             82.33                83.72                 72.33

Key Highlights
--------------
* Benchmarks designed to measure NUMA performance (stream & sc_membench)
  see significant improvements. (This indicates that the new pinning
  options should be working correctly)

* sc_membench latency increases drastically increases for
  1_host_numa_bound. (I am not sure why.)

* Memory latency variance decreases once vCPUs are pinned.
  sc_membench latency standard deviation (ns):
    0: 0.36, 1: 0.35, 2: 0.04, 3: 0.06
   Pinning doesn't just lower latency; it
  makes latency predictable, which matters for tail-sensitive
  workloads.

* sysbench_mem_bw_local throughput increases monotonically with pinning
  specificity:
    0_topology_only:    11173 MiB/s
    1_host_numa_bound:  20742 MiB/s
    2_vcpu_pinned_numa: 21150 MiB/s
    3_vcpu_pinned_1to1: 22892 MiB/s
  95th percentile latency drops in the same order:
    0: 309 ms, 1: 227 ms, 2: 190 ms, 3: 164 ms.
  This is a per-thread workload (each thread allocates its own buffer),
  so the result matches the stream/sc_membench picture: pinning helps
  when memory access is thread-local.

* Memory bandwidth and latency for pinning=numa and pinning=one-to-one
  is generally very similar, with the expection for the
  sysbench_mem_bw (512 MB shared buffer, global scope) case.

  This is a shared-buffer workload. Here pinning=numa drastically
  underperformes and pinning=one-to-one overperforms.

  I cannot give a sensible rationale for why this is.

Verdict
-------
* The new pinning options definitely provides substantial bandwith
  and latancy benefits for certain workloads.

* Specific workloads must be tested, as pinning=numa can
  unexpectedly lead to significant performance decrease, as can be
  observed in the sysbench_mem_bw benchmark.

* Two issues in PVE::QemuServer::Pinning that may contribute to noise:
  - choose_single_cpu() picks (keys %$set)[0] from a Perl hash, whose
    iteration order is randomised per process. Identical qm start runs
    therefore produce different vCPU-to-host-CPU mappings.
  - SMT siblings are not excluded. In one run, guest vCPU 2 was pinned
    to host CPU 21 (SMT sibling of host CPU 5) while all other vCPUs
    landed on primary cores.

    My suggestion on how this could be solved is specified in [7]

References
----------
[1] https://www.cs.virginia.edu/stream/
[2] https://github.com/SpareCores/sc-membench
[3] https://github.com/akopytov/sysbench
[4] https://www.amd.com/en/developer/zen-software-studio/applications/spack/stream-benchmark.html
[5] https://learn.microsoft.com/en-us/azure/high-performance-computing/performance-benchmarking/stream-benchmark
[6] https://arxiv.org/pdf/1103.3225
[7] https://lore.proxmox.com/pve-devel/DLBPNU78N5YH.9QNKITVI57GR@proxmox.com/