From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id DC12D1FF0AB for ; Wed, 09 Sep 2026 15:52:38 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 167A5215D3; Wed, 09 Sep 2026 15:52:36 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Wed, 09 Sep 2026 15:52:28 +0200 Message-Id: Subject: Re: [RFC PATCH qemu-server] fix #7282: allow (NUMA aware) vCPU pinning From: "Elias Huhsovitz" To: "Dominik Csapak" , X-Mailer: aerc 0.20.0 References: <20260217114813.2063770-1-d.csapak@proxmox.com> <86b830f6-da3f-4ebf-82bb-5284594506a9@proxmox.com> In-Reply-To: <86b830f6-da3f-4ebf-82bb-5284594506a9@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1788961939942 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.246 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: IVOFIHAYXDZFML273FUF5O7QJMMAITBI X-Message-ID-Hash: IVOFIHAYXDZFML273FUF5O7QJMMAITBI X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu Mar 12, 2026 at 11:29 AM CET, Dominik Csapak wrote: > I did try to do a few benchmarks methodically but the results are > not really what was expected. > > Either my methodology is wrong (wrong benchmarks, etc.) or > the platform where I tested is actually not suited for this kind of testi= ng. > > Still writing it down, so maybe someone else can give feedback ;) I ran some of my own benchmarks. Comments inline. TL;DR Performance is very mixed. Some scenarios seem to benefit greatly from the option, while others see a decrease in performance. > > The platform was an 1st generation EPYC 7351P 16 Core processor with > 32 threads (not the fastes, but what I had on hand). It was configured > with 64 GB DDR4 memory with 2667MT/s (4x16GB sticks) > > This CPU has 4 NUMA nodes: > numactl -H output: > > available: 4 nodes (0-3) > node 0 cpus: 0 1 2 3 16 17 18 19 > node 0 size: 15936 MB > node 0 free: 14110 MB > node 1 cpus: 4 5 6 7 20 21 22 23 > node 1 size: 16124 MB > node 1 free: 14142 MB > node 2 cpus: 8 9 10 11 24 25 26 27 > node 2 size: 16124 MB > node 2 free: 15210 MB > node 3 cpus: 12 13 14 15 28 29 30 31 > node 3 size: 16080 MB > node 3 free: 13467 MB > node distances: > node 0 1 2 3 > 0: 10 16 16 16 > 1: 16 10 16 16 > 2: 16 16 10 16 > 3: 16 16 16 10 I used the same machine (i.e., same processor & memory) > I ran multiple benchmarks: > > * cpubench1a, which tries to be a general cpu benchmark that > covers a wide array of sub-tests and claims to be NUMA aware[0] > has a single core and multi-core score. > * sysbench in cpu mode (sysbench cpu run) > * sysbench in default memory mode (sysbench memory run) > * sysbench in memory mode with higher block/total size > (sysbench memory run --time=3D60 --memory-total-size=3D500G=20 > --memory-block-size=3D4M) Benchmak Tools -------------- I choose to run * sysbench memory run --threads=3D8 --time=3D30 --memory-block-size=3D1G --= memory-total-size=3D500G * OMP_NUM_THREADS=3D8 OMP_PROC_BIND=3Dspread stream The stream benchmark tools is available at [0], with a basic usage guide at= [1]. I compiled it using the flags: -O2 -fopenmp -DSTREAM_ARRAY_SIZE=3D80000000 I increased the sysbench memory-block-size, since I feared that the memory might not be stressed enough due to the large L3 Cache of the CPU. > This was run in a virtual machine (which was the only thing running > on that host at the time) with 8GB memory with following=20 > cpu/numa/pinning options: > Host System ----------- I ran my benchmarks on: OS: Fedora Linux 44 (Server Edition) x86_64 Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-11.0) Kernel: Linux 7.1.13-200.fc44.x86_64 Memory: 8GB Methodology ----------- 4 Configurations: 1. True baseline: vNUMA enabled, no host node binding, no pinning. 2. Bound baseline: vNUMA enabled, strict host node binding (policy=3Dbind),= no pinning. 3. NUMA pinning: Bound baseline plus pinning=3Dnuma. 4. One-to-one pinning: Bound baseline plus pinning=3Done-to-one. Key Highlights -------------- The policy=3Dbind seems to generate the most significant improvement. * One-to-one pinning performed best in sysbench. It reached 15.9 GB/s throughput (compared to 13.1 GB/s for the bound baseline) * NUMA pinning unexpectedly degraded sysbench performance. Throughput dropped to 9.3 GB/s, and the 95th percentile latency increased to 1561 ms. * STREAM showed different behavior. NUMA pinning achieved the highest bandwidth (53 GB/s), while one-to-one pinning reached 50 GB/s. Results =3D=3D=3D=3D=3D=3D=3D Overview -------- +----------------------------+------------------+-------------------+------= ------------+-------------------+ | Metric | 0 True Baseline | 1 Bound Baseline | 2 NU= MA Pinning | 3 1=E2=80=91to=E2=80=911 Pinning | +----------------------------+------------------+-------------------+------= ------------+-------------------+ | Description | No bind, no pin | Bind, no pin | Bind= + NUMA | Bind + 1=E2=80=91to=E2=80=911 | | STREAM Triad (MB/s) | 34,658.3 | 39,948.5 | = 53,085.5 | 50,040.2 | | Sysbench Throughput (MiB/s)| 11,036.56 | 13,106.62 | = 9,317.09 | 15,931.88 | | Sysbench Avg (ms) | 739.90 | 594.63 | = 871.81 | 481.80 | | Sysbench 95th %ile (ms) | 1,235.62 | 1,109.09 | = 1,561.52 | 877.61 | | Sysbench Max (ms) | 1,825.46 | 2,234.71 | = 2,032.69 | 1,526.76 | +----------------------------+------------------+-------------------+------= ------------+-------------------+ Relevant Output - stream ------------------------ Here is the full test output for the most important benchmarks 1 Bound Baseline Function Best Rate MB/s Avg time Min time Max time Copy: 58175.9 0.024162 0.022002 0.039968 Scale: 37273.1 0.036262 0.034341 0.049191 Add: 40191.0 0.049311 0.047772 0.055433 Triad: 39948.5 0.049245 0.048062 0.052773 2 NUMA Pinning: Function Best Rate MB/s Avg time Min time Max time Copy: 65752.7 0.019580 0.019467 0.020121 Scale: 46764.9 0.027768 0.027371 0.029283 Add: 53321.7 0.036913 0.036008 0.042480 Triad: 53085.5 0.036421 0.036168 0.037505 3 1-to-1 Pinning: Function Best Rate MB/s Avg time Min time Max time Copy: 65140.0 0.019791 0.019650 0.020635 Scale: 44258.3 0.029850 0.028921 0.032909 Add: 49457.8 0.040148 0.038821 0.045838 Triad: 50040.2 0.038486 0.038369 0.038615 Relevant Results - sysbench: ---------------------------- 1 Bound Baseline: Total operations: 388 ( 12.80 per second) 397312.00 MiB transferred (13106.62 MiB/sec) General statistics: total time: 30.3120s total number of events: 388 Latency (ms): min: 231.64 avg: 594.63 max: 2234.71 95th percentile: 1109.09 sum: 230714.98 Threads fairness: events (avg/stddev): 48.5000/8.96 execution time (avg/stddev): 28.8394/2.31 2 NUMA Pinning: Total operations: 278 ( 9.10 per second) 284672.00 MiB transferred (9317.09 MiB/sec) General statistics: total time: 30.5519s total number of events: 278 Latency (ms): min: 333.28 avg: 871.81 max: 2032.69 95th percentile: 1561.52 sum: 242363.64 Threads fairness: events (avg/stddev): 34.7500/15.75 execution time (avg/stddev): 30.2955/0.18 3 1-to-1 Pinning: Total operations: 469 ( 15.56 per second) 480256.00 MiB transferred (15931.88 MiB/sec) General statistics: total time: 30.1426s total number of events: 469 Latency (ms): min: 179.94 avg: 481.80 max: 1526.76 95th percentile: 877.61 sum: 225964.06 Threads fairness: events (avg/stddev): 58.6250/4.09 execution time (avg/stddev): 28.2455/1.85 My current thesis ----------------- It seems to me as though it only helps with workloads where the memory is already distributed across nodes. This would explain why the stream test improves with NUMA, while the sysbench test seems to regress with NUMA Pinning. Perhaps we also need to control memory placement? I will now take a more detailed look at the code and provide my comments in a separate mail. References ---------- [0] https://www.cs.virginia.edu/stream/ [1] https://www.amd.com/en/developer/zen-software-studio/applications/spack= /stream-benchmark.html