From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 26CDF1FF0B3 for ; Fri, 11 Sep 2026 16:34:36 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 099EE214DF; Fri, 11 Sep 2026 16:34:33 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Fri, 11 Sep 2026 16:34:23 +0200 Message-Id: From: "Elias Huhsovitz" To: "Elias Huhsovitz" , "Dominik Csapak" , Subject: Re: [RFC PATCH qemu-server] fix #7282: allow (NUMA aware) vCPU pinning X-Mailer: aerc 0.20.0 References: <20260217114813.2063770-1-d.csapak@proxmox.com> <86b830f6-da3f-4ebf-82bb-5284594506a9@proxmox.com> In-Reply-To: X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1789137253523 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.239 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: M3DIRIOBYONGB33Z22UQJ2EI7Y3JGSDZ X-Message-ID-Hash: M3DIRIOBYONGB33Z22UQJ2EI7Y3JGSDZ X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed Sep 9, 2026 at 3:52 PM CEST, Elias Huhsovitz wrote: > On Thu Mar 12, 2026 at 11:29 AM CET, Dominik Csapak wrote: >> I did try to do a few benchmarks methodically but the results are >> not really what was expected. >> >> Either my methodology is wrong (wrong benchmarks, etc.) or >> the platform where I tested is actually not suited for this kind of test= ing. >> >> Still writing it down, so maybe someone else can give feedback ;) > > I ran some of my own benchmarks. Comments inline. Upon reading the code in more detail and learning more about NUMA in genera= l i have revised my benchmarking methology. Host System ------------- OS: Proxmox VE 9.2.18 x86_64 Kernel: Linux 7.0.14-16-pve CPU: AMD EPYC 7351P 16-Core Processor Memory: 64 GB DDR4 memory with 2667MT/s (4x16GB sticks) numactl -H: available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 16 17 18 19 node 0 size: 15891 MB node 0 free: 14693 MB node 1 cpus: 4 5 6 7 20 21 22 23 node 1 size: 16124 MB node 1 free: 15174 MB node 2 cpus: 8 9 10 11 24 25 26 27 node 2 size: 16124 MB node 2 free: 14838 MB node 3 cpus: 12 13 14 15 28 29 30 31 node 3 size: 16123 MB node 3 free: 14978 MB node distances: node 0 1 2 3 0: 10 16 16 16 1: 16 10 16 16 2: 16 16 10 16 3: 16 16 16 10 Guest System ----------- OS: Fedora Linux 44 (Server Edition) x86_64 Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-11.0) Kernel: Linux 7.1.13-200.fc44.x86_64 Memory: 8GB Benchmarks Tools ---------------- * STREAM [1] (Tool seems to be widely used, see [4,5,6]) compiled via: gcc -O3 -fopenmp -march=3Dnative -mcmodel=3Dmedium -DSTREAM_ARRAY_SIZE=3D16= 0000000 -DNTIMES=3D40 stream.c -o stream executed via (stream): OMP_NUM_THREADS=3D8 OMP_PROC_BIND=3Dspread OMP_PLACES=3Dcores stream (The OMP_NUM_THREADS could theoretically be omitted, since per default OpenMP uses all available threads). * sc-membench [2] compiled via: make numa executed via (sc_membench): OMP_NUM_THREADS=3D8 OMP_PROC_BIND=3Dspread OMP_PLACES=3Dcores membench-numa= -s 262144 -r 3 -t 30 * sysbench [3] executed via (sysbench_mem_bw): sysbench memory --threads=3D8 --time=3D30 --memory-block-size=3D512M --memo= ry-total-size=3D400G --memory-access-mode=3Dseq run executed via (sysbench_mem_bw_local): sysbench memory --threads=3D8 --time=3D30 --memory-block-size=3D512M --memo= ry-total-size=3D400G --memory-access-mode=3Dseq --memory-scope=3Dlocalrun Configurations -------------- * 0_topology_only: guest NUMA defined, no host binding * 1_host_numa_bound: hostnodes=3DN,policy=3Dbind, no vCPU pinning * 2_vcpu_pinned_numa: same 1_host_numa_bound + pinning=3Dnuma * 3_vcpu_pinned_1to1: same as 1_host_numa_bound + pinning=3Done-to-one Results =3D=3D=3D=3D=3D=3D=3D Metric 0_topology_only 1_host_numa_bound = 2_vcpu_pinned_numa 3_vcpu_pinned_1to1 ---------------------------------------------------------------------------= ----------------------------------------- sc_membench: copy MB/s 8388.38 8922.02 = 9078.42 9015.42 sc_membench: largest size KB 262144 262144 = 262144 262144 sc_membench: latency ns 118.26 167.54 = 112.56 113.38 sc_membench: latency sd ns 0.36 0.35 = 0.04 0.06 sc_membench: read MB/s 30015.90 45060.94 = 60461.67 60265.49 sc_membench: write MB/s 12776.54 30013.02 = 30904.18 30731.53 stream: Add MB/s 34690.2 45003.3 = 53325.9 52752.3 stream: Copy MB/s 43383.2 63388.4 = 66914.0 67128.2 stream: Scale MB/s 30353.7 40952.8 = 47707.9 47004.5 stream: Triad MB/s 34527.1 41649.4 = 53401.6 52790.7 sysbench_mem_bw: MiB/sec 14898.10 15583.45 = 10916.81 19167.89 sysbench_mem_bw: lat 95th ms 530.08 484.44 = 746.32 325.98 sysbench_mem_bw: lat avg ms 253.90 238.65 = 344.17 206.83 sysbench_mem_bw_local: MiB/sec 11173.40 20742.30 = 21149.99 22891.73 sysbench_mem_bw_local: lat 95th ms 308.84 227.40 = 189.93 164.45 sysbench_mem_bw_local: lat avg ms 174.89 82.33 = 83.72 72.33 Key Highlights -------------- * Benchmarks designed to measure NUMA performance (stream & sc_membench) see significant improvements. (This indicates that the new pinning options should be working correctly) * sc_membench latency increases drastically increases for 1_host_numa_bound. (I am not sure why.) * Memory latency variance decreases once vCPUs are pinned. sc_membench latency standard deviation (ns): 0: 0.36, 1: 0.35, 2: 0.04, 3: 0.06 Pinning doesn't just lower latency; it makes latency predictable, which matters for tail-sensitive workloads. * sysbench_mem_bw_local throughput increases monotonically with pinning specificity: 0_topology_only: 11173 MiB/s 1_host_numa_bound: 20742 MiB/s 2_vcpu_pinned_numa: 21150 MiB/s 3_vcpu_pinned_1to1: 22892 MiB/s 95th percentile latency drops in the same order: 0: 309 ms, 1: 227 ms, 2: 190 ms, 3: 164 ms. This is a per-thread workload (each thread allocates its own buffer), so the result matches the stream/sc_membench picture: pinning helps when memory access is thread-local. * Memory bandwidth and latency for pinning=3Dnuma and pinning=3Done-to-one is generally very similar, with the expection for the sysbench_mem_bw (512 MB shared buffer, global scope) case. This is a shared-buffer workload. Here pinning=3Dnuma drastically underperformes and pinning=3Done-to-one overperforms. I cannot give a sensible rationale for why this is. Verdict ------- * The new pinning options definitely provides substantial bandwith and latancy benefits for certain workloads. * Specific workloads must be tested, as pinning=3Dnuma can unexpectedly lead to significant performance decrease, as can be observed in the sysbench_mem_bw benchmark. * Two issues in PVE::QemuServer::Pinning that may contribute to noise: - choose_single_cpu() picks (keys %$set)[0] from a Perl hash, whose iteration order is randomised per process. Identical qm start runs therefore produce different vCPU-to-host-CPU mappings. - SMT siblings are not excluded. In one run, guest vCPU 2 was pinned to host CPU 21 (SMT sibling of host CPU 5) while all other vCPUs landed on primary cores. My suggestion on how this could be solved is specified in [7] References ---------- [1] https://www.cs.virginia.edu/stream/ [2] https://github.com/SpareCores/sc-membench [3] https://github.com/akopytov/sysbench [4] https://www.amd.com/en/developer/zen-software-studio/applications/spack= /stream-benchmark.html [5] https://learn.microsoft.com/en-us/azure/high-performance-computing/perf= ormance-benchmarking/stream-benchmark [6] https://arxiv.org/pdf/1103.3225 [7] https://lore.proxmox.com/pve-devel/DLBPNU78N5YH.9QNKITVI57GR@proxmox.co= m/