Re: [RFC PATCH qemu-server] fix #7282: allow (NUMA aware) vCPU pinning

["Elias Huhsovitz" <e.huhsovitz@proxmox.com>]

On Thu Mar 12, 2026 at 11:29 AM CET, Dominik Csapak wrote:
> I did try to do a few benchmarks methodically but the results are
> not really what was expected.
>
> Either my methodology is wrong (wrong benchmarks, etc.) or
> the platform where I tested is actually not suited for this kind of testing.
>
> Still writing it down, so maybe someone else can give feedback ;)

I ran some of my own benchmarks. Comments inline.

TL;DR Performance is very mixed. Some scenarios seem to benefit greatly
from the option, while others see a decrease in performance.

>
> The platform was an 1st generation EPYC 7351P 16 Core processor with
> 32 threads (not the fastes, but what I had on hand). It was configured
> with 64 GB DDR4 memory with 2667MT/s (4x16GB sticks)
>
> This CPU has 4 NUMA nodes:
> numactl -H output:
>
> available: 4 nodes (0-3)
> node 0 cpus: 0 1 2 3 16 17 18 19
> node 0 size: 15936 MB
> node 0 free: 14110 MB
> node 1 cpus: 4 5 6 7 20 21 22 23
> node 1 size: 16124 MB
> node 1 free: 14142 MB
> node 2 cpus: 8 9 10 11 24 25 26 27
> node 2 size: 16124 MB
> node 2 free: 15210 MB
> node 3 cpus: 12 13 14 15 28 29 30 31
> node 3 size: 16080 MB
> node 3 free: 13467 MB
> node distances:
> node     0    1    2    3
>     0:   10   16   16   16
>     1:   16   10   16   16
>     2:   16   16   10   16
>     3:   16   16   16   10

I used the same machine (i.e., same processor & memory)

> I ran multiple benchmarks:
>
> * cpubench1a, which tries to be a general cpu benchmark that
>    covers a wide array of sub-tests and claims to be NUMA aware[0]
>    has a single core and multi-core score.
> * sysbench in cpu mode (sysbench cpu run)
> * sysbench in default memory mode (sysbench memory run)
> * sysbench in memory mode with higher block/total size
>    (sysbench memory run --time=60 --memory-total-size=500G 
> --memory-block-size=4M)

Benchmak Tools
--------------
I choose to run
* sysbench memory run --threads=8 --time=30 --memory-block-size=1G --memory-total-size=500G
* OMP_NUM_THREADS=8 OMP_PROC_BIND=spread stream

The stream benchmark tools is available at [0], with a basic usage guide at [1].
I compiled it using the flags:
-O2 -fopenmp -DSTREAM_ARRAY_SIZE=80000000

I increased the sysbench memory-block-size, since I feared that the
memory might not be stressed enough due to the large L3 Cache of the CPU.

> This was run in a virtual machine (which was the only thing running
> on that host at the time) with 8GB memory with following 
> cpu/numa/pinning options:
>

Host System
-----------
I ran my benchmarks on:
OS: Fedora Linux 44 (Server Edition) x86_64
Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-11.0)
Kernel: Linux 7.1.13-200.fc44.x86_64
Memory: 8GB

Methodology
-----------
4 Configurations:

1. True baseline: vNUMA enabled, no host node binding, no pinning.
2. Bound baseline: vNUMA enabled, strict host node binding (policy=bind), no pinning.
3. NUMA pinning: Bound baseline plus pinning=numa.
4. One-to-one pinning: Bound baseline plus pinning=one-to-one.

Key Highlights
--------------
The policy=bind seems to generate the most significant improvement.

* One-to-one pinning performed best in sysbench. It reached 15.9 GB/s
  throughput (compared to 13.1 GB/s for the bound baseline)
* NUMA pinning unexpectedly degraded sysbench performance. Throughput
  dropped to 9.3 GB/s, and the 95th percentile latency increased
  to 1561 ms.
* STREAM showed different behavior. NUMA pinning achieved the highest
  bandwidth (53 GB/s), while one-to-one pinning reached 50 GB/s.

Results
=======
Overview
--------
+----------------------------+------------------+-------------------+------------------+-------------------+
| Metric                     | 0 True Baseline  | 1 Bound Baseline   | 2 NUMA Pinning  | 3 1‑to‑1 Pinning  |
+----------------------------+------------------+-------------------+------------------+-------------------+
| Description                | No bind, no pin  | Bind, no pin       | Bind + NUMA     | Bind + 1‑to‑1     |
| STREAM Triad (MB/s)        |        34,658.3  |         39,948.5   |        53,085.5 |         50,040.2  |
| Sysbench Throughput (MiB/s)|       11,036.56  |        13,106.62   |        9,317.09 |        15,931.88  |
| Sysbench Avg (ms)          |          739.90  |           594.63   |          871.81 |           481.80  |
| Sysbench 95th %ile (ms)    |        1,235.62  |         1,109.09   |        1,561.52 |           877.61  |
| Sysbench Max (ms)          |        1,825.46  |         2,234.71   |        2,032.69 |         1,526.76  |
+----------------------------+------------------+-------------------+------------------+-------------------+

Relevant Output - stream
------------------------
Here is the full test output for the most important benchmarks

1 Bound Baseline
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           58175.9     0.024162     0.022002     0.039968
Scale:          37273.1     0.036262     0.034341     0.049191
Add:            40191.0     0.049311     0.047772     0.055433
Triad:          39948.5     0.049245     0.048062     0.052773

2 NUMA Pinning:
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           65752.7     0.019580     0.019467     0.020121
Scale:          46764.9     0.027768     0.027371     0.029283
Add:            53321.7     0.036913     0.036008     0.042480
Triad:          53085.5     0.036421     0.036168     0.037505

3 1-to-1 Pinning:
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           65140.0     0.019791     0.019650     0.020635
Scale:          44258.3     0.029850     0.028921     0.032909
Add:            49457.8     0.040148     0.038821     0.045838
Triad:          50040.2     0.038486     0.038369     0.038615

Relevant Results - sysbench:
----------------------------
1 Bound Baseline:

Total operations: 388 (   12.80 per second)

397312.00 MiB transferred (13106.62 MiB/sec)

General statistics:
    total time:                          30.3120s
    total number of events:              388

Latency (ms):
         min:                                  231.64
         avg:                                  594.63
         max:                                 2234.71
         95th percentile:                     1109.09
         sum:                               230714.98

Threads fairness:
    events (avg/stddev):           48.5000/8.96
    execution time (avg/stddev):   28.8394/2.31

2 NUMA Pinning:
Total operations: 278 (    9.10 per second)

284672.00 MiB transferred (9317.09 MiB/sec)

General statistics:
    total time:                          30.5519s
    total number of events:              278

Latency (ms):
         min:                                  333.28
         avg:                                  871.81
         max:                                 2032.69
         95th percentile:                     1561.52
         sum:                               242363.64

Threads fairness:
    events (avg/stddev):           34.7500/15.75
    execution time (avg/stddev):   30.2955/0.18

3 1-to-1 Pinning:

Total operations: 469 (   15.56 per second)

480256.00 MiB transferred (15931.88 MiB/sec)

General statistics:
    total time:                          30.1426s
    total number of events:              469

Latency (ms):
         min:                                  179.94
         avg:                                  481.80
         max:                                 1526.76
         95th percentile:                      877.61
         sum:                               225964.06

Threads fairness:
    events (avg/stddev):           58.6250/4.09
    execution time (avg/stddev):   28.2455/1.85

My current thesis
-----------------
It seems to me as though it only helps with workloads where the memory
is already distributed across nodes. This would explain why the stream
test improves with NUMA, while the sysbench test seems to regress with
NUMA Pinning.

Perhaps we also need to control memory placement?

I will now take a more detailed look at the code and provide my comments
in a separate mail.

References
----------
[0] https://www.cs.virginia.edu/stream/
[1] https://www.amd.com/en/developer/zen-software-studio/applications/spack/stream-benchmark.html