From: Dominik Csapak <d.csapak@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu-server v4 13/14] machine: add 'pci-topology' option
Date: Fri, 13 Nov 2026 15:39:29 +0100 [thread overview]
Message-ID: <20261113143939.3012964-14-d.csapak@proxmox.com> (raw)
In-Reply-To: <20261113143939.3012964-1-d.csapak@proxmox.com>
Guest software sometimes uses the PCI topology to choose CPUs and
DMA paths. Since every hostpci device with pcie=1 gets a root port on
the root bus, the guest can't see which devices are close to which NUMA
node or to each other.
The new option selects how these devices are placed in the guest:
* 'flat': the current behavior and the default
* 'numa': puts the devices behind an expander bridge for the guest
NUMA node that is bound to their host NUMA node
* 'host': additionally replicates the bridges that more than one
device is behind on the host as switches
For that, read the NUMA node and the bridges of each device from sysfs
and pass them to the bindings. This is done after choosing the host
devices, since only then it's known which ones are used. For mediated
devices, the actual device is only chosen when starting the guest, so
only use what all alternatives have in common.
The guest must have NUMA enabled with a guest node bound to the host
node of each device, as the expander bridges can only be assigned to
existing guest NUMA nodes. The bindings only support the topologies
with the 'grouped' PCI layout.
Since the placement depends on the hardware of the host, prevent live
migration for all topologies except 'flat'.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
src/PVE/QemuMigrate.pm | 6 +
src/PVE/QemuServer/Machine.pm | 20 +++
src/PVE/QemuServer/PCI.pm | 118 +++++++++++++++++-
src/test/QemuServerTest/CommandLineMocks.pm | 21 ++++
...-grouped-layout-hostpci-host-topology.conf | 19 +++
...uped-layout-hostpci-host-topology.conf.cmd | 57 +++++++++
6 files changed, 238 insertions(+), 3 deletions(-)
create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf
create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd
diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm
index 8176afe5..476719bc 100644
--- a/src/PVE/QemuMigrate.pm
+++ b/src/PVE/QemuMigrate.pm
@@ -268,6 +268,12 @@ sub prepare {
}
}
+ my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine});
+ if (my $topology = $machine_conf->{'pci-topology'}) {
+ die "can't live migrate running VM with PCI topology '$topology'\n"
+ if $running && $topology ne 'flat';
+ }
+
my ($loc_res, $mapped_res, $missing_mappings_by_node) =
PVE::QemuMigrate::Helpers::check_local_resources($conf, $running, 1);
my $blocking_resources = [];
diff --git a/src/PVE/QemuServer/Machine.pm b/src/PVE/QemuServer/Machine.pm
index f30c1ef3..01673be0 100644
--- a/src/PVE/QemuServer/Machine.pm
+++ b/src/PVE/QemuServer/Machine.pm
@@ -111,6 +111,26 @@ my $machine_fmt = {
. " its own set of bridges. Changing it changes the addresses of all devices. This"
. " option is experimental.",
},
+ 'pci-topology' => {
+ type => 'string',
+ optional => 1,
+ default => 'flat',
+ enum => [qw(flat numa host)],
+ description =>
+ "Determines how hostpci devices with pcie set are laid out on the guest.",
+ verbose_description => <<EODESC,
+This option changes how hostpci devices with pcie set are passed through to the
+guest. Note that all values except the default prevent live migration and
+require the 'grouped' PCI layout.
+
+* flat: no special handling and no NUMA node assignment. The default.
+* numa: each device is attached to the guest NUMA node that is mapped to its
+ host NUMA node. Fails if the guest NUMA nodes don't match the host ones.
+* host: the overall switch/bridge structure is replicated. Devices behind a
+ switch on the host are behind a switch in the guest as well. This requires
+ the guest NUMA nodes to match as well.
+EODESC
+ },
};
PVE::JSONSchema::register_format('pve-qemu-machine-fmt', $machine_fmt);
diff --git a/src/PVE/QemuServer/PCI.pm b/src/PVE/QemuServer/PCI.pm
index 338f2d1a..56d12bb4 100644
--- a/src/PVE/QemuServer/PCI.pm
+++ b/src/PVE/QemuServer/PCI.pm
@@ -3,9 +3,11 @@ package PVE::QemuServer::PCI;
use warnings;
use strict;
+use Cwd qw(abs_path);
use Fcntl qw(SEEK_SET);
use IO::File;
+use PVE::File;
use PVE::GuestHelpers;
use PVE::JSONSchema;
use PVE::Mapping::PCI;
@@ -15,6 +17,7 @@ use PVE::Tools;
use PVE::QemuServer::Helpers;
use PVE::QemuServer::Machine;
+use PVE::QemuServer::Memory;
use PVE::QemuServer::PCI::Mdev;
use base 'Exporter';
@@ -366,6 +369,100 @@ sub choose_hostpci_devices {
return $devices;
}
+# Returns the addresses of all bridges between the host bridge and the given host device, starting
+# at the root port. A PCIe switch shows up as its upstream and downstream port.
+sub get_host_pci_bridges {
+ my ($pciid) = @_;
+
+ # resolves to e.g. /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.0
+ my $sysfs_path = "/sys/bus/pci/devices/$pciid";
+ die "unable to resolve sysfs path of PCI device '$pciid'\n" if !-e $sysfs_path;
+ my $path = abs_path($sysfs_path);
+
+ my @bridges;
+ for my $part (split('/', $path)) {
+ push @bridges, $1 if $part =~ m/^($PCIRE)$/; # filter relevant parts and untaint
+ }
+ pop @bridges; # remove the device itself
+
+ return \@bridges;
+}
+
+# Returns the host NUMA node the given host device is connected to.
+sub get_host_numa_node {
+ my ($pciid) = @_;
+
+ my $node = PVE::File::file_read_first_line("/sys/bus/pci/devices/$pciid/numa_node");
+
+ # hosts with only one node report -1
+ return defined($node) && $node =~ m/^(\d+)$/ ? int($1) : 0;
+}
+
+# Returns where the chosen hostpci devices with pcie=1 are connected on the host, as needed to
+# place them in the guest with the 'numa' and 'host' PCI topologies.
+sub get_hostpcie_placements {
+ my ($conf, $pci_devices, $topology) = @_;
+
+ return [] if !scalar(keys $pci_devices->%*);
+
+ # the expander bridges can only be assigned to existing guest NUMA nodes
+ die "PCI topology '$topology' requires NUMA to be enabled\n" if !$conf->{numa};
+
+ # we need a clear host to guest numa node mapping
+ my $guest_to_host = PVE::QemuServer::Memory::get_numa_guest_to_host_map($conf);
+ my $host_to_guest = {};
+ for my $guest_node (sort { $a <=> $b } keys $guest_to_host->%*) {
+ my $host_node = $guest_to_host->{$guest_node};
+ next if !defined($host_node) || $host_node !~ m/^\d+$/;
+ $host_to_guest->{$host_node} //= $guest_node;
+ }
+
+ my $placements = [];
+ for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
+ my $d = $pci_devices->{"hostpci$i"};
+ next if !$d || !$d->{pcie};
+
+ # For mediated devices, the actual device is only chosen when starting the guest, so only
+ # use numa nodes/bridges that all alternatives have in common.
+ my @pciids = map { $_->{id} } $d->{ids}->@*;
+ @pciids = ($pciids[0]) if !$d->{mdev};
+
+ my ($host_node, $bridges);
+ for my $pciid (@pciids) {
+ my $node = get_host_numa_node($pciid);
+ $host_node //= $node;
+ die "hostpci$i: PCI topology '$topology' requires all alternative devices to be on"
+ . " the same host NUMA node\n"
+ if $node != $host_node;
+
+ next if $topology ne 'host';
+
+ my $current = get_host_pci_bridges($pciid);
+ if (!defined($bridges)) {
+ $bridges = $current;
+ next;
+ }
+ # keep the bridges shared by all alternatives
+ my $common = [];
+ for (my $j = 0; $j < scalar(@$bridges) && $j < scalar(@$current); $j++) {
+ last if $bridges->[$j] ne $current->[$j];
+ push @$common, $bridges->[$j];
+ }
+ $bridges = $common;
+ }
+
+ my $guest_node = $host_to_guest->{$host_node};
+ die "hostpci$i: no guest NUMA node is bound to host NUMA node $host_node\n"
+ if !defined($guest_node);
+
+ my $placement = { index => $i, 'numa-node' => int($guest_node) };
+ $placement->{switches} = $bridges if defined($bridges);
+ push $placements->@*, $placement;
+ }
+
+ return $placements;
+}
+
sub print_hostpci_devices {
my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = @_;
@@ -376,6 +473,15 @@ sub print_hostpci_devices {
my $pciaddr;
my $pci_devices = choose_hostpci_devices(parse_hostpci_devices($conf), $vmid, $dry_run);
+ # the placements can only be derived after we chose the pci devices
+ my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine});
+ my $topology = $machine_conf->{'pci-topology'} // 'flat';
+ my $placements;
+ if ($topology ne 'flat') {
+ $placements = get_hostpcie_placements($conf, $pci_devices, $topology);
+ push @$devices, $hw_config->hostpcie_bus_devices($placements)->@*;
+ }
+
for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
my $id = "hostpci$i";
my $d = $pci_devices->{$id};
@@ -384,8 +490,7 @@ sub print_hostpci_devices {
$legacy_igd = 1 if $d->{'legacy-igd'};
if (my $pcie = $d->{pcie}) {
- # the root ports come from pci_bus_devices
- $pciaddr = $hw_config->addr("hostpcie$i");
+ $pciaddr = $hw_config->addr("hostpcie$i", $placements);
} else {
my $pci_name = $d->{'legacy-igd'} ? 'legacy-igd' : $id;
$pciaddr = $hw_config->addr($pci_name);
@@ -711,6 +816,12 @@ sub hardware_config {
$mainboard = "i440fx";
}
+ my $topology = $machine_conf->{'pci-topology'} // 'flat';
+ my $layout = $machine_conf->{'pci-layout'} // 'legacy';
+
+ die "PCI topology '$topology' requires the 'grouped' PCI layout\n"
+ if $topology ne 'flat' && $layout eq 'legacy';
+
my ($major, $minor, $patch, $pve) = PVE::QemuServer::Helpers::version_parts($machine_version);
return PVE::RS::PCI::HardwareConfig->new({
@@ -729,8 +840,9 @@ sub hardware_config {
},
'legacy-igd' => $legacy_igd,
'ostype' => $conf->{ostype} // 'other',
- 'layout' => $machine_conf->{'pci-layout'} // 'legacy',
+ 'layout' => $layout,
'hostpcie-devices' => $hostpcie,
+ 'hostpci-topology' => $topology,
});
}
diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm
index 0ae5e6b5..361f1f9a 100644
--- a/src/test/QemuServerTest/CommandLineMocks.pm
+++ b/src/test/QemuServerTest/CommandLineMocks.pm
@@ -168,6 +168,15 @@ my $pci_devs = [
"0000:02:00.5",
];
+# where the devices on a host bus are connected: the NUMA node and the bridges above them,
+# starting at the root port. Buses not listed here are on node 0 without any bridge.
+my $pci_host_topology = {
+ '01' => { node => 0, bridges => ['0000:00:01.0'] },
+ '07' => { node => 0, bridges => ['0000:00:03.0'] },
+ # behind a switch
+ 'd0' => { node => 1, bridges => ['0000:c0:01.0', '0000:ce:00.0', '0000:cf:08.0'] },
+};
+
my $pci_map_config = {
ids => {
someGpu => {
@@ -620,6 +629,18 @@ $pci_module->mock(
create_nvidia_device => sub {
die "create_nvidia_device should not be called for 'qm showcmd'\n";
},
+ get_host_numa_node => sub {
+ my ($pciid) = @_;
+
+ my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
+ return $pci_host_topology->{$bus}->{node} // 0;
+ },
+ get_host_pci_bridges => sub {
+ my ($pciid) = @_;
+
+ my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
+ return [($pci_host_topology->{$bus}->{bridges} // [])->@*];
+ },
);
1;
diff --git a/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf
new file mode 100644
index 00000000..db2efa90
--- /dev/null
+++ b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf
@@ -0,0 +1,19 @@
+# TEST: Config with q35, NUMA, grouped pci-layout & hostpci passthrough with host pci-topology
+bios: ovmf
+bootdisk: scsi0
+cores: 1
+efidisk0: local:100/vm-100-disk-1.qcow2,size=128K
+hostpci0: d0:13.0,pcie=1
+hostpci1: d0:15.1,pcie=1
+hostpci2: mapping=someNic,pcie=1
+hostpci3: mapping=someGpu,mdev=some-model,pcie=1
+hostpci4: 00:f4.0
+machine: q35,pci-layout=grouped,pci-topology=host
+memory: 512
+net0: virtio=2E:01:68:F9:9C:87,bridge=vmbr0
+numa: 1
+ostype: l26
+scsihw: virtio-scsi-pci
+smbios1: uuid=3dd750ce-d910-44d0-9493-525c0be4e687
+sockets: 2
+vmgenid: 54d1c06c-8f5b-440f-b5b2-6eab1380e13d
diff --git a/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd
new file mode 100644
index 00000000..f1f9b6e6
--- /dev/null
+++ b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd
@@ -0,0 +1,57 @@
+/usr/bin/kvm \
+ -id 8006 \
+ -name vm8006 \
+ -no-shutdown \
+ -chardev 'socket,id=qmp,path=/var/run/qemu-server/8006.qmp,server=on,wait=off' \
+ -object 'monitor-qmp,chardev=qmp,id=monitor-qmp' \
+ -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' \
+ -object 'monitor-qmp,chardev=qmp-event,id=monitor-qmp-event' \
+ -pidfile /var/run/qemu-server/8006.pid \
+ -daemonize \
+ -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \
+ -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+ -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+ -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","read-only":false,"throttle-group":"throttle-drive-efidisk0"}' \
+ -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \
+ -smp '2,sockets=2,cores=1,maxcpus=2' \
+ -nodefaults \
+ -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
+ -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
+ -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
+ -m 512 \
+ -object 'memory-backend-ram,id=ram-node0,size=256M' \
+ -numa 'node,nodeid=0,cpus=0,memdev=ram-node0' \
+ -object 'memory-backend-ram,id=ram-node1,size=256M' \
+ -numa 'node,nodeid=1,cpus=1,memdev=ram-node1' \
+ -global 'ICH9-LPC.disable_s3=1' \
+ -global 'ICH9-LPC.disable_s4=1' \
+ -device 'pcie-pci-bridge,id=pve.sys0,bus=pcie.0,addr=0x4' \
+ -device 'pcie-pci-bridge,id=pve.virtio-fs0,bus=pcie.0,addr=0x5' \
+ -device 'pcie-pci-bridge,id=pve.net0,bus=pcie.0,addr=0x6,multifunction=on' \
+ -device 'pcie-pci-bridge,id=pve.net1,bus=pcie.0,addr=0x6.1' \
+ -device 'pcie-pci-bridge,id=pve.scsi0,bus=pcie.0,addr=0x7' \
+ -device 'pcie-pci-bridge,id=pve.virtio-blk0,bus=pcie.0,addr=0x8' \
+ -device 'pcie-pci-bridge,id=pve.hostpci0,bus=pcie.0,addr=0x9' \
+ -device 'vmgenid,guid=54d1c06c-8f5b-440f-b5b2-6eab1380e13d' \
+ -device 'qemu-xhci,p2=15,p3=15,id=xhci,bus=pve.sys0,addr=0x3' \
+ -device 'usb-hub,id=hub,bus=xhci.0,port=15' \
+ -device 'usb-tablet,id=tablet,bus=xhci.0,port=15.7' \
+ -device 'pxb-pcie,id=pve.numa0,bus=pcie.0,addr=0x10,bus_nr=32,numa_node=0' \
+ -device 'pcie-root-port,id=pve.root-port2,bus=pve.numa0,addr=0x3,port=2,chassis=128,slot=2,x-speed=16,x-width=32' \
+ -device 'pcie-root-port,id=pve.root-port3,bus=pve.numa0,addr=0x4,port=3,chassis=128,slot=3,x-speed=16,x-width=32' \
+ -device 'pxb-pcie,id=pve.numa1,bus=pcie.0,addr=0x11,bus_nr=48,numa_node=1' \
+ -device 'pcie-root-port,id=pve.switch-uplink0,bus=pve.numa1,addr=0x1,port=0,chassis=129,slot=0,x-speed=16,x-width=32' \
+ -device 'x3130-upstream,id=pve.switch0,bus=pve.switch-uplink0,addr=0x0' \
+ -device 'xio3130-downstream,id=pve.switch-port0,bus=pve.switch0,addr=0x0,port=0,chassis=144,slot=0' \
+ -device 'xio3130-downstream,id=pve.switch-port1,bus=pve.switch0,addr=0x1,port=1,chassis=144,slot=1' \
+ -device 'vfio-pci,host=0000:d0:13.0,id=hostpci0,bus=pve.switch-port0,addr=0x0' \
+ -device 'vfio-pci,host=0000:d0:15.1,id=hostpci1,bus=pve.switch-port1,addr=0x0' \
+ -device 'vfio-pci,host=0000:07:10.0,id=hostpci2,bus=pve.root-port2,addr=0x0' \
+ -device 'vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000003-0000-0000-0000-000000008006,id=hostpci3,bus=pve.root-port3,addr=0x0' \
+ -device 'vfio-pci,host=0000:00:f4.0,id=hostpci4,bus=pve.hostpci0,addr=0x5' \
+ -device 'VGA,id=vga,bus=pcie.0,addr=0x2' \
+ -device 'virtio-balloon-pci,id=balloon0,bus=pve.sys0,addr=0x2,free-page-reporting=on' \
+ -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
+ -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
+ -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pve.net0,addr=0x1,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300,host_mtu=1500,host_tunnel=off' \
+ -machine 'pflash0=pflash0,pflash1=drive-efidisk0,hpet=off,type=q35+pve0'
--
2.47.3
next prev parent reply other threads:[~2026-11-13 14:41 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 14:39 [PATCH manager/proxmox-perl-rs/qemu-server/qemu-server-rs v4 00/14] pci-handling rewrite Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 01/14] add pve-qemu-server-pci crate for guest PCI address generation Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 02/14] pci: legacy: generate pci bridges Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 03/14] pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 04/14] pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 05/14] pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 06/14] pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-13 14:39 ` [PATCH proxmox-perl-rs v4 07/14] pve: add bindings for `pve-qemu-server-pci` crate Dominik Csapak
2026-11-13 14:39 ` [PATCH proxmox-perl-rs v4 08/14] pve: pci: add host placements for the hostpci topology Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 09/14] pci: use PVE::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 10/14] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 11/14] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 12/14] machine: add 'pci-layout' option Dominik Csapak
2026-11-13 14:39 ` Dominik Csapak [this message]
2026-11-13 14:39 ` [PATCH pve-manager v4 14/14] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113143939.3012964-14-d.csapak@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox