[pve-devel] [PATCH docs/http-server/manager 0/4] fix #5392: pveproxy, pvedaemon: make number of worker processes configurable

[pve-devel] [PATCH docs/http-server/manager 0/4] fix #5392: pveproxy, pvedaemon: make number of worker processes configurable

[Friedrich Weber]

From [1]: For pveproxy and pvedaemon, max_workers is currently hardcoded to 3
in PVE::Service::{pveproxy,pvedaemon}. This may not be enough for
automation-heavy workloads that trigger a lot of API requests that are
synchronously handled by pveproxy or pvedaemon, see e.g. #5391. This was also
encountered occasionally in enterprise support.

changes since rfc:
- add limit on acceptable MAX_WORKERS setting (thx Thomas!)
- add pve-docs patch

rfc: https://lore.proxmox.com/pve-devel/20250729113523.72332-1-f.weber@proxmox.com/T/

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5392

http-server:

Friedrich Weber (1):
  api server: proxy config: read MAX_WORKERS integer key

 src/PVE/APIServer/Utils.pm | 8 ++++++++
 1 file changed, 8 insertions(+)


manager:

Friedrich Weber (2):
  partially fix #5392: pveproxy: make number of workers configurable
  partially fix #5392: pvedaemon: make number of workers configurable

 PVE/Service/pvedaemon.pm | 6 +++++-
 PVE/Service/pveproxy.pm  | 3 ++-
 2 files changed, 7 insertions(+), 2 deletions(-)


docs:

Friedrich Weber (1):
  pveproxy, pvedaemon: document MAX_WORKERS setting

 pvedaemon.adoc | 17 +++++++++++++++++
 pveproxy.adoc  | 18 ++++++++++++++++++
 2 files changed, 35 insertions(+)


Summary over all repositories:
  5 files changed, 50 insertions(+), 2 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH http-server 1/1] api server: proxy config: read MAX_WORKERS integer key

[Friedrich Weber]

Read the MAX_WORKERS value in /etc/default/<proxyname>. If it is not
an integer between 0 and 128, ignore and warn.

The lower limit was chosen because at least one worker process must
exist. The upper limit was chosen because more than 127 worker
processes should not be necessary and a positive impact on performance
is doubtful. If this limit turns out to be too low, it can still be
extended in the future.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    changes since rfc:
    - add lower and upper limit (thx Thomas!)

 src/PVE/APIServer/Utils.pm | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/PVE/APIServer/Utils.pm b/src/PVE/APIServer/Utils.pm
index 1430c98..4cceb71 100644
--- a/src/PVE/APIServer/Utils.pm
+++ b/src/PVE/APIServer/Utils.pm
@@ -28,6 +28,7 @@ sub read_proxy_config {
     $shcmd .= 'echo \"DISABLE_TLS_1_3:\$DISABLE_TLS_1_3\";';
     $shcmd .= 'echo \"PROXY_REAL_IP_HEADER:\$PROXY_REAL_IP_HEADER\";';
     $shcmd .= 'echo \"PROXY_REAL_IP_ALLOW_FROM:\$PROXY_REAL_IP_ALLOW_FROM\";';
+    $shcmd .= 'echo \"MAX_WORKERS:\$MAX_WORKERS\";';
 
     my $data = -f $conffile ? `bash -c "$shcmd"` : '';
 
@@ -77,6 +78,13 @@ sub read_proxy_config {
                 push @$ips, Net::IP->new(normalize_v4_in_v6($ip)) || die Net::IP::Error() . "\n";
             }
             $res->{$key} = $ips;
+        } elsif ($key eq 'MAX_WORKERS') {
+            if ($value =~ /^\d+$/ && $value > 0 && $value < 128) {
+                $res->{$key} = int($value);
+            } else {
+                warn "MAX_WORKERS specified in $conffile is not an integer between"
+                    ." 0 and 128: $value\n";
+            }
         } elsif (grep { $key eq $_ } @$boolean_options) {
             die "unknown value '$value' - use 0 or 1\n" if $value !~ m/^(0|1)$/;
             $res->{$key} = $value;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH http-server 1/1] api server: proxy config: read MAX_WORKERS integer key

[Thomas Lamprecht]

On Tue, 29 Jul 2025 17:50:55 +0200, Friedrich Weber wrote:
> Read the MAX_WORKERS value in /etc/default/<proxyname>. If it is not
> an integer between 0 and 128, ignore and warn.
> 
> The lower limit was chosen because at least one worker process must
> exist. The upper limit was chosen because more than 127 worker
> processes should not be necessary and a positive impact on performance
> is doubtful. If this limit turns out to be too low, it can still be
> extended in the future.
> 
> [...]

Applied, thanks!

[1/1] api server: proxy config: read MAX_WORKERS integer key
      commit: d72947f112536b9ae024f48a88eb368d8209f5f9


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 1/2] partially fix #5392: pveproxy: make number of workers configurable

[Friedrich Weber]

The number of pveproxy worker processes is currently hardcoded to 3.
This may not be enough for automation-heavy workloads that trigger a
lot of API requests that are synchronously handled by pveproxy.

Hence, allow specifying MAX_WORKERS in /etc/default/pveproxy to
override the number of workers.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    I decided against setting max_workers directly directly in
    %daemon_options, to avoid having to call read_proxy_config already
    then. If I understand correctly, overriding $self->{max_workers} in
    init should be fine because it's only used in PVE::Daemon's
    $server_run after init was called.
    
    changes since rfc:
    none

 PVE/Service/pveproxy.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/PVE/Service/pveproxy.pm b/PVE/Service/pveproxy.pm
index c4bb54ea..ab455ff9 100755
--- a/PVE/Service/pveproxy.pm
+++ b/PVE/Service/pveproxy.pm
@@ -29,7 +29,7 @@ use base qw(PVE::Daemon);
 my $cmdline = [$0, @ARGV];
 
 my %daemon_options = (
-    max_workers => 3,
+    max_workers => 3, # may be overridden in init
     restart_on_error => 5,
     stop_wait_time => 15,
     leave_children_open_on_reload => 1,
@@ -64,6 +64,7 @@ sub init {
 
     # we use same ALLOW/DENY/POLICY as pveproxy
     my $proxyconf = PVE::APIServer::Utils::read_proxy_config($self->{name});
+    $self->{max_workers} = $proxyconf->{MAX_WORKERS} if $proxyconf->{MAX_WORKERS};
 
     my $accept_lock_fn = "/var/lock/pveproxy.lck";
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager 1/2] partially fix #5392: pveproxy: make number of workers configurable

[Thomas Lamprecht]

On Tue, 29 Jul 2025 17:50:56 +0200, Friedrich Weber wrote:
> The number of pveproxy worker processes is currently hardcoded to 3.
> This may not be enough for automation-heavy workloads that trigger a
> lot of API requests that are synchronously handled by pveproxy.
> 
> Hence, allow specifying MAX_WORKERS in /etc/default/pveproxy to
> override the number of workers.
> 
> [...]

Applied, thanks!

[1/2] partially fix #5392: pveproxy: make number of workers configurable
      commit: d7fec8f9fffc9a9405c497092da36cbc8f506bf8


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 2/2] partially fix #5392: pvedaemon: make number of workers configurable

[Friedrich Weber]

The number of pvedaemon worker processes is currently hardcoded to 3.
This may not be enough for automation-heavy workloads that trigger a
lot of API requests that are synchronously handled by pvedaemon.

Hence, read /etc/default/pvedaemon when starting pvedaemon and allow
overriding the number of workers by specifying MAX_WORKERS in this
file. All other values are only relevant for pveproxy/spiceproxy and
thus ignored.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    I'm not sure if reading /etc/default/pvedaemon, which was not used
    until now, is a good solution here, but I decided to go for it because
    it seems relatively straightforward and analogous to pveproxy.
    
    Also here, I decided against setting max_workers directly directly in
    %daemon_options, to avoid having to call read_proxy_config already
    then. If I understand correctly, overriding $self->{max_workers} in
    init should be fine because it's only used in PVE::Daemon's
    $server_run after init was called.
    
    changes since rfc:
    none

 PVE/Service/pvedaemon.pm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/PVE/Service/pvedaemon.pm b/PVE/Service/pvedaemon.pm
index 9d7cbc0f..4493f234 100755
--- a/PVE/Service/pvedaemon.pm
+++ b/PVE/Service/pvedaemon.pm
@@ -15,7 +15,7 @@ use base qw(PVE::Daemon);
 my $cmdline = [$0, @ARGV];
 
 my %daemon_options = (
-    max_workers => 3,
+    max_workers => 3, # may be overridden in init
     restart_on_error => 5,
     stop_wait_time => 15,
     leave_children_open_on_reload => 1,
@@ -26,6 +26,10 @@ my $daemon = __PACKAGE__->new('pvedaemon', $cmdline, %daemon_options);
 sub init {
     my ($self) = @_;
 
+    # all options other than MAX_WORKERS are ignored
+    my $proxyconf = PVE::APIServer::Utils::read_proxy_config($self->{name});
+    $self->{max_workers} = $proxyconf->{MAX_WORKERS} if $proxyconf->{MAX_WORKERS};
+
     my $accept_lock_fn = "/var/lock/pvedaemon.lck";
 
     my $lockfh = IO::File->new(">>${accept_lock_fn}")
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager 2/2] partially fix #5392: pvedaemon: make number of workers configurable

[Thomas Lamprecht]

On Tue, 29 Jul 2025 17:50:57 +0200, Friedrich Weber wrote:
> The number of pvedaemon worker processes is currently hardcoded to 3.
> This may not be enough for automation-heavy workloads that trigger a
> lot of API requests that are synchronously handled by pvedaemon.
> 
> Hence, read /etc/default/pvedaemon when starting pvedaemon and allow
> overriding the number of workers by specifying MAX_WORKERS in this
> file. All other values are only relevant for pveproxy/spiceproxy and
> thus ignored.
> 
> [...]

Applied, thanks!

[2/2] partially fix #5392: pvedaemon: make number of workers configurable
      commit: 934305ea1def085ee138788cc6192d52fc075cbe


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH docs 1/1] pveproxy, pvedaemon: document MAX_WORKERS setting

[Friedrich Weber]

For pveproxy, add it to the description of settings that can be
adjusted in /etc/default/pveproxy.

For pvedaemon, this is currently the only setting that can be adjusted
in /etc/default/pvedaemon.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---
 pvedaemon.adoc | 17 +++++++++++++++++
 pveproxy.adoc  | 18 ++++++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/pvedaemon.adoc b/pvedaemon.adoc
index 66c751a..d1923a4 100644
--- a/pvedaemon.adoc
+++ b/pvedaemon.adoc
@@ -30,6 +30,23 @@ NOTE: The daemon listens to a local address only, so you cannot access
 it from outside. The `pveproxy` daemon exposes the API to the outside
 world.
 
+[[pvedaemon_max_workers]]
+Number of Workers
+-----------------
+
+`pvedaemon` delegates handling of incoming requests to worker processes. By
+default, `pvedaemon` spawns 3 worker processes, which is sufficient for most
+workloads. For automation-heavy workloads that issue a huge volume of API
+requests and that experience slow request handling or timeouts, the number of
+worker processes can be increased by setting `MAX_WORKERS` in
+`/etc/default/pvedaemon`, for example:
+
+ MAX_WORKERS=5
+
+Note that a higher number of worker processes may result in higher CPU usage.
+The number of worker processes must be greater than 0 and smaller than 128.
+
+The same setting exists for xref:pveproxy_max_workers[pveproxy].
 
 ifdef::manvolnum[]
 include::pve-copyright.adoc[]
diff --git a/pveproxy.adoc b/pveproxy.adoc
index a311afd..6999b4e 100644
--- a/pveproxy.adoc
+++ b/pveproxy.adoc
@@ -227,6 +227,24 @@ IP addresses can be specified using any syntax understood by `Net::IP`. The
 name `all` is an alias for `0/0` and `::/0` (meaning all IPv4 and IPv6
 addresses).
 
+[[pveproxy_max_workers]]
+Number of Workers
+-----------------
+
+`pveproxy` delegates handling of incoming requests to worker processes. By
+default, `pveproxy` spawns 3 worker processes, which is sufficient for most
+workloads. For automation-heavy workloads that issue a huge volume of API
+requests and that experience slow request handling or timeouts, the number of
+worker processes can be increased by setting `MAX_WORKERS` in
+`/etc/default/pveproxy`, for example:
+
+ MAX_WORKERS=5
+
+Note that a higher number of worker processes may result in higher CPU usage.
+The number of worker processes must be greater than 0 and smaller than 128.
+
+The same setting exists for xref:pvedaemon_max_workers[pvedaemon].
+
 ifdef::manvolnum[]
 include::pve-copyright.adoc[]
 endif::manvolnum[]
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH docs 1/1] pveproxy, pvedaemon: document MAX_WORKERS setting

[Thomas Lamprecht]

On Tue, 29 Jul 2025 17:50:58 +0200, Friedrich Weber wrote:
> For pveproxy, add it to the description of settings that can be
> adjusted in /etc/default/pveproxy.
> 
> For pvedaemon, this is currently the only setting that can be adjusted
> in /etc/default/pvedaemon.
> 
> 
> [...]

Applied, thanks!

[1/1] pveproxy, pvedaemon: document MAX_WORKERS setting
      commit: 7153da5a9edf2284e159c85f7fc3fb29f45c6ded


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [RFC http-server/manager 0/3] fix #5392: pveproxy, pvedaemon: make number of worker processes configurable

[Friedrich Weber]

From [1]: For pveproxy and pvedaemon, max_workers is currently hardcoded to 3
in PVE::Service::{pveproxy,pvedaemon}. This may not be enough for
automation-heavy workloads that trigger a lot of API requests that are
synchronously handled by pveproxy or pvedaemon, see e.g. #5391. This was also
encountered occasionally in enterprise support.

Marking as RFC, as

- I'm not sure if the feature is a good idea and the implementation is acceptable
- Documentation is currently missing. If this RFC is deemed acceptable, I'd
  include it in the v1.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5392

http-server:

Friedrich Weber (1):
  api server: proxy config: read MAX_WORKERS integer key

 src/PVE/APIServer/Utils.pm | 7 +++++++
 1 file changed, 7 insertions(+)


manager:

Friedrich Weber (2):
  partially fix #5392: pveproxy: make number of workers configurable
  partially fix #5392: pvedaemon: make number of workers configurable

 PVE/Service/pvedaemon.pm | 6 +++++-
 PVE/Service/pveproxy.pm  | 3 ++-
 2 files changed, 7 insertions(+), 2 deletions(-)


Summary over all repositories:
  3 files changed, 14 insertions(+), 2 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 1/2] partially fix #5392: pveproxy: make number of workers configurable

[Friedrich Weber]

The number of pveproxy worker processes is currently hardcoded to 3.
This may not be enough for automation-heavy workloads that trigger a
lot of API requests that are synchronously handled by pveproxy.

Hence, allow specifying MAX_WORKERS in /etc/default/pveproxy to
override the number of workers.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    I decided against setting max_workers directly directly in
    %daemon_options, to avoid having to call read_proxy_config already
    then. If I understand correctly, overriding $self->{max_workers} in
    init should be fine because it's only used in PVE::Daemon's
    $server_run after init was called.

 PVE/Service/pveproxy.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/PVE/Service/pveproxy.pm b/PVE/Service/pveproxy.pm
index c4bb54ea..ab455ff9 100755
--- a/PVE/Service/pveproxy.pm
+++ b/PVE/Service/pveproxy.pm
@@ -29,7 +29,7 @@ use base qw(PVE::Daemon);
 my $cmdline = [$0, @ARGV];
 
 my %daemon_options = (
-    max_workers => 3,
+    max_workers => 3, # may be overridden in init
     restart_on_error => 5,
     stop_wait_time => 15,
     leave_children_open_on_reload => 1,
@@ -64,6 +64,7 @@ sub init {
 
     # we use same ALLOW/DENY/POLICY as pveproxy
     my $proxyconf = PVE::APIServer::Utils::read_proxy_config($self->{name});
+    $self->{max_workers} = $proxyconf->{MAX_WORKERS} if $proxyconf->{MAX_WORKERS};
 
     my $accept_lock_fn = "/var/lock/pveproxy.lck";
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel