[pve-devel] [PATCH 0/5] fix #3593: Added vm core pinning

[pve-devel] [PATCH 0/5] fix #3593: Added vm core pinning

[Daniel Bowder]

These five patches add the ability to pin a QEMU VMs processes to a defined set of CPU cores. The changes required spanned multiple repos and have all been included in these five patches. The patches add "cpuset" as an option in the qm configuration file and enable the editing of this parameter via the gui.

pve-docs
 qm.conf.5-opts.adoc | 4 ++++
 1 file changed, 4 insertions(+)

pve-guest-common
 src/PVE/GuestHelpers.pm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

qemu-server
 PVE/QemuServer.pm | 49 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

proxmox-widget-toolkit
 src/Toolkit.js | 5 +++++
 src/Utils.js   | 2 ++
 2 files changed, 7 insertions(+)

pve-manager
 www/manager6/qemu/Options.js | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

[pve-devel] [PATCH 1/5] fix #3593: Added vm core pinning pve-docs

[Daniel Bowder]

Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
---
 The first patch adds an entry to the qm conf adding a new option: cpuset. The cpuset here is the same cpuset used by the taskset application for pinning a process to a cpu core. This can be found in `man cpuset`, or https://linux.die.net/man/7/cpuset 
 qm.conf.5-opts.adoc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/qm.conf.5-opts.adoc b/qm.conf.5-opts.adoc
index a56dc5d..992cd51 100644
--- a/qm.conf.5-opts.adoc
+++ b/qm.conf.5-opts.adoc
@@ -155,6 +155,10 @@ Limit of CPU usage.
 +
 NOTE: If the computer has 2 CPUs, it has total of '2' CPU time. Value '0' indicates no CPU limit.
 
+`cpuset`: `<string>`::
+
+Set of CPU cores to pin the virtual machine processes to. This is a comma sepparated list of numbers or ranges in list format as defined by the Linux man page for cpuset. ( e.g `0,4-6,9` )
+
 `cpuunits`: `<integer> (1 - 262144)` ('default =' `cgroup v1: 1024, cgroup v2: 100`)::
 
 CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, the more CPU time this VM gets. Number is relative to weights of all the other running VMs.
-- 
2.30.2

Re: [pve-devel] [PATCH 1/5] fix #3593: Added vm core pinning pve-docs

[Matthias Heiserer]

On 08.06.2022 13:54, Daniel Bowder wrote:
> Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
> ---
>   The first patch adds an entry to the qm conf adding a new option: cpuset. The cpuset here is the same cpuset used by the taskset application for pinning a process to a cpu core. This can be found in `man cpuset`, or https://linux.die.net/man/7/cpuset
>   qm.conf.5-opts.adoc | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/qm.conf.5-opts.adoc b/qm.conf.5-opts.adoc
> index a56dc5d..992cd51 100644
> --- a/qm.conf.5-opts.adoc
> +++ b/qm.conf.5-opts.adoc
> @@ -155,6 +155,10 @@ Limit of CPU usage.
>   +
>   NOTE: If the computer has 2 CPUs, it has total of '2' CPU time. Value '0' indicates no CPU limit.
>   
> +`cpuset`: `<string>`::
> +
> +Set of CPU cores to pin the virtual machine processes to. This is a comma sepparated list of numbers or ranges in list format as defined by the Linux man page for cpuset. ( e.g `0,4-6,9` )
typo: "separated"

Personally, I'd prefer something alone the lines of "[..] of numbers in 
the cpuset `List format`", but no hard feelings.

> +
>   `cpuunits`: `<integer> (1 - 262144)` ('default =' `cgroup v1: 1024, cgroup v2: 100`)::
>   
>   CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, the more CPU time this VM gets. Number is relative to weights of all the other running VMs.

[pve-devel] [PATCH 2/5] fix #3593: Added vm core pinning pve-guest-common

[Daniel Bowder]

Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
---
 The second patch adds a utility function to obtain the pid of the VM, then calls taskset on that pid with the cpuset in the qm conf. This execution of taskset pins the VMs process to the defined cpu cores. This utility function acts similarly to the exec_hookscript utility function. In fact, in the third patch this utility function will be called just after the "post-start" call to hookscript.
 src/PVE/GuestHelpers.pm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/src/PVE/GuestHelpers.pm b/src/PVE/GuestHelpers.pm
index 0fe3fd6..a8f54fa 100644
--- a/src/PVE/GuestHelpers.pm
+++ b/src/PVE/GuestHelpers.pm
@@ -82,6 +82,30 @@ sub guest_migration_lock {
     return $res;
 }
 
+sub exec_taskset {
+	my ($conf, $vmid) = @_;
+
+	return if !$conf->{cpuset};
+
+	eval {
+		# Obtain the vm PID from /run/qemu-server/$vmid.pid
+		PVE::Tools::run_command(
+			[ "cat", "/run/qemu-server/$vmid.pid" ],
+			outfunc => sub {
+				my $line = shift;
+				if ($line =~ m/^([0-9]+)$/) {
+					# Pin the PID to the cpuset
+					PVE::Tools::run_command(
+						[ "taskset", "--cpu-list", "--all-tasks", "--pid", $conf->{cpuset}, $1 ]
+					);
+				} else {
+					warn "cpuset error: failed to obtain vm process id.\n";
+				}
+			}
+		);
+	};
+}
+
 sub check_hookscript {
     my ($volid, $storecfg) = @_;
 
-- 
2.30.2

[pve-devel] [PATCH 3/5] fix #3593: Added vm core pinning qemu-server

[Daniel Bowder]

Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
---
 The third patch adds cpuset as a valid object in the qm conf file. A new type is created called 'pve-cpuset' so that the cpuset can go through some validation before passing it to the taskset command. The exec_taskset command is executed just after the 'post-start' hookscript, which ensures that there is a valid PID to pin.
 PVE/QemuServer.pm | 49 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index e9aa248..6b9abc0 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -713,6 +713,11 @@ EODESCR
 	description => "Some (read-only) meta-information about this guest.",
 	optional => 1,
     },
+	cpuset => {
+	type => 'string', format => 'pve-cpuset',
+	description => "Specifies the cpu core numbers to pin the vm qemu processes to with 'taskset'.",
+	optional => 1,
+	},
 };
 
 my $cicustom_fmt = {
@@ -5802,6 +5807,8 @@ sub vm_start_nolock {
 
     PVE::GuestHelpers::exec_hookscript($conf, $vmid, 'post-start');
 
+    PVE::GuestHelpers::exec_taskset($conf, $vmid);
+
     return $res;
 }
 
@@ -8214,4 +8221,46 @@ sub check_volume_storage_type {
     return 1;
 }
 
+sub parse_cpuset {
+    my ($data) = @_;
+
+    my $res = "";
+
+	# Parse cpuset value
+	foreach my $value (PVE::Tools::split_list($data)) {
+		if ($value =~ m/^([0-9]+)-([0-9]+)$/) {
+			if (int($1) > int($2)) {
+				die "invalid cpuset value '$value', left value must be <= right\n";
+			}
+			my @range = (int($1) .. int($2));
+			for my $cpu (@range) {
+				$res .= "," . $cpu;
+			}
+		} elsif ($value =~ m/^([0-9]+)$/) {
+			$res .= "," . $1;
+		} else {
+			die "invalid cpuset value '$value'\n";
+		}
+	}
+
+	if (!$res) {
+		die "invalid cpuset '$data'\n";
+	}
+
+	# Trim leading ","
+    $res = substr($res, 1);
+    return $res;
+}
+
+PVE::JSONSchema::register_format('pve-cpuset', \&pve_verify_cpuset);
+sub pve_verify_cpuset {
+    my ($value, $noerr) = @_;
+
+    return $value if parse_cpuset($value);
+
+    return if $noerr;
+
+    die "unable to parse cpuset option\n";
+}
+
 1;
-- 
2.30.2

Re: [pve-devel] [PATCH 3/5] fix #3593: Added vm core pinning qemu-server

[Matthias Heiserer]

I'm not sure where it comes from, but when starting a VM, the task log 
receives a bunch of messages on changed affinity:

pid 95121's current affinity list: 0-7
pid 95121's new affinity list: 2,3
pid 95122's current affinity list: 0-7
pid 95122's new affinity list: 2,3
pid 95123's current affinity list: 0-7
pid 95123's new affinity list: 2,3
pid 95184's current affinity list: 0-7
pid 95184's new affinity list: 2,3
pid 95185's current affinity list: 0-7
pid 95185's new affinity list: 2,3
pid 95187's current affinity list: 0-7
pid 95187's new affinity list: 2,3

Can we avoid showing them? I don't think they offer any insight, because 
as I understand it, when starting, the current affinity list is always 
all available cores.

On 08.06.2022 13:54, Daniel Bowder wrote:
> Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
> ---
>   The third patch adds cpuset as a valid object in the qm conf file. A new type is created called 'pve-cpuset' so that the cpuset can go through some validation before passing it to the taskset command. The exec_taskset command is executed just after the 'post-start' hookscript, which ensures that there is a valid PID to pin.
The commit message should be above the lines :)
Otherwise, it won't be part of the commit, but only a comment visible on 
the mailing list.
>   PVE/QemuServer.pm | 49 +++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 49 insertions(+)
> 
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index e9aa248..6b9abc0 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -713,6 +713,11 @@ EODESCR
>   	description => "Some (read-only) meta-information about this guest.",
>   	optional => 1,
>       },
> +	cpuset => {
> +	type => 'string', format => 'pve-cpuset',
> +	description => "Specifies the cpu core numbers to pin the vm qemu processes to with 'taskset'.",
> +	optional => 1,
> +	},
>   };
>   
>   my $cicustom_fmt = {
> @@ -5802,6 +5807,8 @@ sub vm_start_nolock {
>   
>       PVE::GuestHelpers::exec_hookscript($conf, $vmid, 'post-start');
>   
> +    PVE::GuestHelpers::exec_taskset($conf, $vmid);
> +
>       return $res;
>   }
>   
> @@ -8214,4 +8221,46 @@ sub check_volume_storage_type {
>       return 1;
>   }
>   
> +sub parse_cpuset {
> +    my ($data) = @_;
> +
> +    my $res = "";
> +
> +	# Parse cpuset value
> +	foreach my $value (PVE::Tools::split_list($data)) {
> +		if ($value =~ m/^([0-9]+)-([0-9]+)$/) {
> +			if (int($1) > int($2)) {
> +				die "invalid cpuset value '$value', left value must be <= right\n";
> +			}
> +			my @range = (int($1) .. int($2));
> +			for my $cpu (@range) {
> +				$res .= "," . $cpu;
> +			}
> +		} elsif ($value =~ m/^([0-9]+)$/) {
> +			$res .= "," . $1;
> +		} else {
> +			die "invalid cpuset value '$value'\n";
> +		}
> +	}
> +
> +	if (!$res) {
> +		die "invalid cpuset '$data'\n";
> +	}
> +
> +	# Trim leading ","
> +    $res = substr($res, 1);
> +    return $res;
> +}
> +
> +PVE::JSONSchema::register_format('pve-cpuset', \&pve_verify_cpuset);
> +sub pve_verify_cpuset {
> +    my ($value, $noerr) = @_;
> +
> +    return $value if parse_cpuset($value);
> +
> +    return if $noerr;
> +
> +    die "unable to parse cpuset option\n";
> +}
> +
>   1;

BTW, my colleagues will also review your code at some point, so be 
prepared that there might be some more required changes

[pve-devel] [PATCH 4/5] fix #3593: Added vm core pinning proxmox-widget-toolkit

[Daniel Bowder]

Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
---
 The fourth patch adds a javascript type for CPUSet. This type can use a regex match to (mostly) ensure the user types a correct cpuset. There are some fringe cases where the CPUSet can pass the CPUSet_match, but then fail the checks in patch three. (e.g. 0,5-4 will pass, but is invalid). This is not a problem, because the user cannot save this invalid CPUSet per the checks in patch three.
 src/Toolkit.js | 5 +++++
 src/Utils.js   | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/src/Toolkit.js b/src/Toolkit.js
index a1d291e..41673a6 100644
--- a/src/Toolkit.js
+++ b/src/Toolkit.js
@@ -121,6 +121,11 @@ Ext.apply(Ext.form.field.VTypes, {
     },
     HttpProxyText: gettext('Example') + ": http://username:password&#64;host:port/",
 
+    CPUSet: function(v) {
+        return Proxmox.Utils.CPUSet_match.test(v);
+    },
+    CPUSetText: gettext('This is not a valid CPU Set. (e.g. 0,2-6,8)'),
+
     DnsName: function(v) {
 	return Proxmox.Utils.DnsName_match.test(v);
     },
diff --git a/src/Utils.js b/src/Utils.js
index 6a03057..8f1c839 100644
--- a/src/Utils.js
+++ b/src/Utils.js
@@ -1315,6 +1315,8 @@ utilities: {
 	me.DnsName_match = new RegExp("^" + DnsName_REGEXP + "$");
 	me.DnsName_or_Wildcard_match = new RegExp("^(?:\\*\\.)?" + DnsName_REGEXP + "$");
 
+	me.CPUSet_match = /^(?:(?:[0-9]+,)|(?:[0-9]+-[0-9]+,))*(?:(?:[0-9]+)$|(?:[0-9]+-[0-9]+)$)/;
+
 	me.HostPort_match = new RegExp("^(" + IPV4_REGEXP + "|" + DnsName_REGEXP + ")(?::(\\d+))?$");
 	me.HostPortBrackets_match = new RegExp("^\\[(" + IPV6_REGEXP + "|" + IPV4_REGEXP + "|" + DnsName_REGEXP + ")\\](?::(\\d+))?$");
 	me.IP6_dotnotation_match = new RegExp("^(" + IPV6_REGEXP + ")(?:\\.(\\d+))?$");
-- 
2.30.2

[pve-devel] [PATCH 5/5] fix #3593: Added vm core pinning pve-manager

[Daniel Bowder]

Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
---
 The fifth patch adds the cpuset value to the GUI under the VM.Config.Options panel. The cpuset is set as a vtype of CPUSet so that it can be checked by the regex match in the fouth patch. This was modeled after the existing 'name' option, where the labels and textfield type have been changed.
 www/manager6/qemu/Options.js | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/www/manager6/qemu/Options.js b/www/manager6/qemu/Options.js
index a1def4bb..ce356130 100644
--- a/www/manager6/qemu/Options.js
+++ b/www/manager6/qemu/Options.js
@@ -341,6 +341,36 @@ Ext.define('PVE.qemu.Options', {
 	    hookscript: {
 		header: gettext('Hookscript'),
 	    },
+		cpuset: {
+		required: false,
+		header: gettext('CPU Set'),
+		defaultValue: "",
+		editor: caps.vms['VM.Config.Options'] ? {
+		    xtype: 'proxmoxWindowEdit',
+		    subject: gettext('CPU Set'),
+		    items: {
+			xtype: 'inputpanel',
+			items: {
+			    xtype: 'textfield',
+			    name: 'cpuset',
+				vtype: 'CPUSet',
+			    value: '',
+			    fieldLabel: gettext('cpuset'),
+			    allowBlank: true,
+				emptyText: gettext("Pin to cores (e.g. 0,2-6,8)"),
+			},
+			onGetValues: function(values) {
+			    var params = values;
+			    if (values.cpuset === undefined ||
+				values.cpuset === null ||
+				values.cpuset === '') {
+				params = { 'delete': 'cpuset' };
+			    }
+			    return params;
+			},
+		    },
+		} : undefined,
+		},
 	};
 
 	var baseurl = 'nodes/' + nodename + '/qemu/' + vmid + '/config';
-- 
2.30.2

Re: [pve-devel] [PATCH 5/5] fix #3593: Added vm core pinning pve-manager

[Matthias Heiserer]

On 08.06.2022 13:54, Daniel Bowder wrote:
> Signed-off-by: Daniel Bowder <daniel@bowdernet.com>
> ---
>   The fifth patch adds the cpuset value to the GUI under the VM.Config.Options panel. The cpuset is set as a vtype of CPUSet so that it can be checked by the regex match in the fouth patch. This was modeled after the existing 'name' option, 
typo: fouth -> fourth

Don't think you should be referencing other patches by number in the 
commit message though, so the comment is a fitting location for this remark.
where the labels and textfield type have been changed.
>   www/manager6/qemu/Options.js | 30 ++++++++++++++++++++++++++++++
>   1 file changed, 30 insertions(+)
> 
> diff --git a/www/manager6/qemu/Options.js b/www/manager6/qemu/Options.js
> index a1def4bb..ce356130 100644
> --- a/www/manager6/qemu/Options.js
> +++ b/www/manager6/qemu/Options.js
> @@ -341,6 +341,36 @@ Ext.define('PVE.qemu.Options', {
>   	    hookscript: {
>   		header: gettext('Hookscript'),
>   	    },
> +		cpuset: {
> +		required: false,
> +		header: gettext('CPU Set'),
> +		defaultValue: "",
> +		editor: caps.vms['VM.Config.Options'] ? {
> +		    xtype: 'proxmoxWindowEdit',
> +		    subject: gettext('CPU Set'),
> +		    items: {
> +			xtype: 'inputpanel',
> +			items: {
> +			    xtype: 'textfield',
> +			    name: 'cpuset',
> +				vtype: 'CPUSet',
> +			    value: '',
> +			    fieldLabel: gettext('cpuset'),
> +			    allowBlank: true,
> +				emptyText: gettext("Pin to cores (e.g. 0,2-6,8)"),
> +			},
> +			onGetValues: function(values) {
> +			    var params = values;
> +			    if (values.cpuset === undefined ||
> +				values.cpuset === null ||
> +				values.cpuset === '') {
> +				params = { 'delete': 'cpuset' };
> +			    }
> +			    return params;
> +			},
> +		    },
> +		} : undefined,
> +		},
>   	};
>   
>   	var baseurl = 'nodes/' + nodename + '/qemu/' + vmid + '/config';

Is a string the best way of entering the CPUs in the GUI? Maybe a 
dropdown where you can (un)select the cores?

Re: [pve-devel] [PATCH 0/5] fix #3593: Added vm core pinning

[Matthias Heiserer]

On 08.06.2022 13:54, Daniel Bowder wrote:
> These five patches add the ability to pin a QEMU VMs processes to a defined set of CPU cores. The changes required spanned multiple repos and have all been included in these five patches. The patches add "cpuset" as an option in the qm configuration file and enable the editing of this parameter via the gui.
> 

Thank you for your patches!

Regarding the formatting: Please make sure to follow our Perl and 
Javascript style guides [0][1] and the Developer Documentation [2]. All 
your patches use a nonconformant indentation :/
Also, the repository (e.g. qemu-server) should be part of the prefix, 
next to "PATCH", this makes it easier to parse where they belong.

In case you haven't done so yet, we require a CLA (explained in [2]) 
from you in order to use your code.


[0] https://pve.proxmox.com/wiki/Perl_Style_Guide
[1] https://pve.proxmox.com/wiki/Javascript_Style_Guide
[2] https://pve.proxmox.com/wiki/Developer_Documentation

Re: [pve-devel] [PATCH 0/5] fix #3593: Added vm core pinning

[Daniel Bowder]

> Thank you for your patches!
You are welcome.

> Regarding the formatting: Please make sure to follow our Perl and
> Javascript style guides [0][1] and the Developer Documentation [2]. All
> your patches use a nonconformant indentation :/
I do apologize for that. I read [0] and [2] in their entirety multiple
times. I was very careful to make sure I didn't have any lines that were >
100 chars and I tried to follow the formatting guidelines. I'll need to fix
my editor to match the indentation specifications.

> Also, the repository (e.g. qemu-server) should be part of the prefix,
> next to "PATCH", this makes it easier to parse where they belong.
The instructions on [2] under #Preparing_Patches are a little ambiguous
when it comes to changes to multiple repos that are all related.
Specifically when making changes to multiple repos for one related task.

# cd pve-manager; git format-patch -s -o /tmp/patchq -1
# cd ../pve-guest-common; git format-patch -s -o /tmp/patchq -1
# cd ../pve-docs; git format-patch -s -o /tmp/patchq -1
# git send-email --compose --to=pve-devel@lists.proxmox.com /tmp/patchq/*

I didn't follow these completely blindly, but, I also wasn't 100% certain
what to change for a multi-repo review vs  a single repo review. I'll make
sure to include [ PATCH xxxx a/b ] for each xxxx repo patch in the future.

> In case you haven't done so yet, we require a CLA (explained in [2])
> from you in order to use your code.
I sent in the CLA just before submitting the patches.


I'm somewhat of a noob when it comes to mailing lists as opposed to working
off of branches in a repo. Do I resubmit my patches in the same way that I
did previously after I've resolved the indentation issues?

On Wed, Jun 8, 2022 at 6:00 AM Matthias Heiserer <m.heiserer@proxmox.com>
wrote:

> On 08.06.2022 13:54, Daniel Bowder wrote:
> > These five patches add the ability to pin a QEMU VMs processes to a
> defined set of CPU cores. The changes required spanned multiple repos and
> have all been included in these five patches. The patches add "cpuset" as
> an option in the qm configuration file and enable the editing of this
> parameter via the gui.
> >
>
> Thank you for your patches!
>
> Regarding the formatting: Please make sure to follow our Perl and
> Javascript style guides [0][1] and the Developer Documentation [2]. All
> your patches use a nonconformant indentation :/
> Also, the repository (e.g. qemu-server) should be part of the prefix,
> next to "PATCH", this makes it easier to parse where they belong.
>
> In case you haven't done so yet, we require a CLA (explained in [2])
> from you in order to use your code.
>
>
> [0] https://pve.proxmox.com/wiki/Perl_Style_Guide
> [1] https://pve.proxmox.com/wiki/Javascript_Style_Guide
> [2] https://pve.proxmox.com/wiki/Developer_Documentation
>
>

Re: [pve-devel] [PATCH 0/5] fix #3593: Added vm core pinning

[Matthias Heiserer]

On 08.06.2022 15:21, Daniel Bowder wrote:
>> Thank you for your patches!
> You are welcome.
> 
>> Regarding the formatting: Please make sure to follow our Perl and
>> Javascript style guides [0][1] and the Developer Documentation [2]. All
>> your patches use a nonconformant indentation :/
> I do apologize for that. I read [0] and [2] in their entirety multiple
> times. I was very careful to make sure I didn't have any lines that were >
> 100 chars and I tried to follow the formatting guidelines. I'll need to fix
> my editor to match the indentation specifications.
No worries, the formatting is a bit unusual.

> 
>> Also, the repository (e.g. qemu-server) should be part of the prefix,
>> next to "PATCH", this makes it easier to parse where they belong.
> The instructions on [2] under #Preparing_Patches are a little ambiguous
> when it comes to changes to multiple repos that are all related.
> Specifically when making changes to multiple repos for one related task.
> 
> # cd pve-manager; git format-patch -s -o /tmp/patchq -1
> # cd ../pve-guest-common; git format-patch -s -o /tmp/patchq -1
> # cd ../pve-docs; git format-patch -s -o /tmp/patchq -1
> # git send-email --compose --to=pve-devel@lists.proxmox.com /tmp/patchq/*
> 
> I didn't follow these completely blindly, but, I also wasn't 100% certain
> what to change for a multi-repo review vs  a single repo review. I'll make
> sure to include [ PATCH xxxx a/b ] for each xxxx repo patch in the future.
> 
>> In case you haven't done so yet, we require a CLA (explained in [2])
>> from you in order to use your code.
> I sent in the CLA just before submitting the patches.

Great!>
> I'm somewhat of a noob when it comes to mailing lists as opposed to working
> off of branches in a repo. Do I resubmit my patches in the same way that I
> did previously after I've resolved the indentation issues?
> 
Yes, exactly! Make sure to also add a version in the commit message(e.g: 
"[pve-devel] [PATCH v2 pve-manager 4/5] fix #3593"), and describe which 
changes were made since the last version. You can take a look at other 
mails so see how it is done: 
https://lists.proxmox.com/pipermail/pve-devel/, and in the Developer 
Documentation in the section "Preparing Patches".

I suggest you wait a bit before sending in a v2, I'll first test your patch.


> On Wed, Jun 8, 2022 at 6:00 AM Matthias Heiserer <m.heiserer@proxmox.com>
> wrote:
> 
>> On 08.06.2022 13:54, Daniel Bowder wrote:
>>> These five patches add the ability to pin a QEMU VMs processes to a
>> defined set of CPU cores. The changes required spanned multiple repos and
>> have all been included in these five patches. The patches add "cpuset" as
>> an option in the qm configuration file and enable the editing of this
>> parameter via the gui.
>>>
>>
>> Thank you for your patches!
>>
>> Regarding the formatting: Please make sure to follow our Perl and
>> Javascript style guides [0][1] and the Developer Documentation [2]. All
>> your patches use a nonconformant indentation :/
>> Also, the repository (e.g. qemu-server) should be part of the prefix,
>> next to "PATCH", this makes it easier to parse where they belong.
>>
>> In case you haven't done so yet, we require a CLA (explained in [2])
>> from you in order to use your code.
>>
>>
>> [0] https://pve.proxmox.com/wiki/Perl_Style_Guide
>> [1] https://pve.proxmox.com/wiki/Javascript_Style_Guide
>> [2] https://pve.proxmox.com/wiki/Developer_Documentation
>>
>>
> 

Re: [pve-devel] [PATCH 0/5] fix #3593: Added vm core pinning

[Daniel Bowder]

> Yes, exactly! Make sure to also add a version in the commit message(e.g:
> "[pve-devel] [PATCH v2 pve-manager 4/5] fix #3593"), and describe which
> changes were made since the last version. You can take a look at other
> mails so see how it is done:
> https://lists.proxmox.com/pipermail/pve-devel/, and in the Developer
> Documentation in the section "Preparing Patches".

> I suggest you wait a bit before sending in a v2, I'll first test your
patch.

Sounds good. I will:
1. read up on Preparing Patches
2. correct my formatting settings
3. wait to hear your feedback for incorporation with v2

On Wed, Jun 8, 2022 at 6:32 AM Matthias Heiserer <m.heiserer@proxmox.com>
wrote:

> On 08.06.2022 15:21, Daniel Bowder wrote:
> >> Thank you for your patches!
> > You are welcome.
> >
> >> Regarding the formatting: Please make sure to follow our Perl and
> >> Javascript style guides [0][1] and the Developer Documentation [2]. All
> >> your patches use a nonconformant indentation :/
> > I do apologize for that. I read [0] and [2] in their entirety multiple
> > times. I was very careful to make sure I didn't have any lines that were
> >
> > 100 chars and I tried to follow the formatting guidelines. I'll need to
> fix
> > my editor to match the indentation specifications.
> No worries, the formatting is a bit unusual.
>
> >
> >> Also, the repository (e.g. qemu-server) should be part of the prefix,
> >> next to "PATCH", this makes it easier to parse where they belong.
> > The instructions on [2] under #Preparing_Patches are a little ambiguous
> > when it comes to changes to multiple repos that are all related.
> > Specifically when making changes to multiple repos for one related task.
> >
> > # cd pve-manager; git format-patch -s -o /tmp/patchq -1
> > # cd ../pve-guest-common; git format-patch -s -o /tmp/patchq -1
> > # cd ../pve-docs; git format-patch -s -o /tmp/patchq -1
> > # git send-email --compose --to=pve-devel@lists.proxmox.com
> /tmp/patchq/*
> >
> > I didn't follow these completely blindly, but, I also wasn't 100% certain
> > what to change for a multi-repo review vs  a single repo review. I'll
> make
> > sure to include [ PATCH xxxx a/b ] for each xxxx repo patch in the
> future.
> >
> >> In case you haven't done so yet, we require a CLA (explained in [2])
> >> from you in order to use your code.
> > I sent in the CLA just before submitting the patches.
>
> Great!>
> > I'm somewhat of a noob when it comes to mailing lists as opposed to
> working
> > off of branches in a repo. Do I resubmit my patches in the same way that
> I
> > did previously after I've resolved the indentation issues?
> >
> Yes, exactly! Make sure to also add a version in the commit message(e.g:
> "[pve-devel] [PATCH v2 pve-manager 4/5] fix #3593"), and describe which
> changes were made since the last version. You can take a look at other
> mails so see how it is done:
> https://lists.proxmox.com/pipermail/pve-devel/, and in the Developer
> Documentation in the section "Preparing Patches".
>
> I suggest you wait a bit before sending in a v2, I'll first test your
> patch.
>
>
> > On Wed, Jun 8, 2022 at 6:00 AM Matthias Heiserer <m.heiserer@proxmox.com
> >
> > wrote:
> >
> >> On 08.06.2022 13:54, Daniel Bowder wrote:
> >>> These five patches add the ability to pin a QEMU VMs processes to a
> >> defined set of CPU cores. The changes required spanned multiple repos
> and
> >> have all been included in these five patches. The patches add "cpuset"
> as
> >> an option in the qm configuration file and enable the editing of this
> >> parameter via the gui.
> >>>
> >>
> >> Thank you for your patches!
> >>
> >> Regarding the formatting: Please make sure to follow our Perl and
> >> Javascript style guides [0][1] and the Developer Documentation [2]. All
> >> your patches use a nonconformant indentation :/
> >> Also, the repository (e.g. qemu-server) should be part of the prefix,
> >> next to "PATCH", this makes it easier to parse where they belong.
> >>
> >> In case you haven't done so yet, we require a CLA (explained in [2])
> >> from you in order to use your code.
> >>
> >>
> >> [0] https://pve.proxmox.com/wiki/Perl_Style_Guide
> >> [1] https://pve.proxmox.com/wiki/Javascript_Style_Guide
> >> [2] https://pve.proxmox.com/wiki/Developer_Documentation
> >>
> >>
> >
>
>
>