[pve-devel] [RFC http-server/manager 0/3] fix #5392: pveproxy, pvedaemon: make number of worker processes configurable

[pve-devel] [RFC http-server/manager 0/3] fix #5392: pveproxy, pvedaemon: make number of worker processes configurable

[Friedrich Weber]

From [1]: For pveproxy and pvedaemon, max_workers is currently hardcoded to 3
in PVE::Service::{pveproxy,pvedaemon}. This may not be enough for
automation-heavy workloads that trigger a lot of API requests that are
synchronously handled by pveproxy or pvedaemon, see e.g. #5391. This was also
encountered occasionally in enterprise support.

Marking as RFC, as

- I'm not sure if the feature is a good idea and the implementation is acceptable
- Documentation is currently missing. If this RFC is deemed acceptable, I'd
  include it in the v1.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5392

http-server:

Friedrich Weber (1):
  api server: proxy config: read MAX_WORKERS integer key

 src/PVE/APIServer/Utils.pm | 7 +++++++
 1 file changed, 7 insertions(+)


manager:

Friedrich Weber (2):
  partially fix #5392: pveproxy: make number of workers configurable
  partially fix #5392: pvedaemon: make number of workers configurable

 PVE/Service/pvedaemon.pm | 6 +++++-
 PVE/Service/pveproxy.pm  | 3 ++-
 2 files changed, 7 insertions(+), 2 deletions(-)


Summary over all repositories:
  3 files changed, 14 insertions(+), 2 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH http-server 1/1] api server: proxy config: read MAX_WORKERS integer key

[Friedrich Weber]

Read the MAX_WORKERS value in /etc/default/<proxyname>. If it is not
an integer, ignore and warn.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---
 src/PVE/APIServer/Utils.pm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/PVE/APIServer/Utils.pm b/src/PVE/APIServer/Utils.pm
index 1430c98..f2c4892 100644
--- a/src/PVE/APIServer/Utils.pm
+++ b/src/PVE/APIServer/Utils.pm
@@ -28,6 +28,7 @@ sub read_proxy_config {
     $shcmd .= 'echo \"DISABLE_TLS_1_3:\$DISABLE_TLS_1_3\";';
     $shcmd .= 'echo \"PROXY_REAL_IP_HEADER:\$PROXY_REAL_IP_HEADER\";';
     $shcmd .= 'echo \"PROXY_REAL_IP_ALLOW_FROM:\$PROXY_REAL_IP_ALLOW_FROM\";';
+    $shcmd .= 'echo \"MAX_WORKERS:\$MAX_WORKERS\";';
 
     my $data = -f $conffile ? `bash -c "$shcmd"` : '';
 
@@ -77,6 +78,12 @@ sub read_proxy_config {
                 push @$ips, Net::IP->new(normalize_v4_in_v6($ip)) || die Net::IP::Error() . "\n";
             }
             $res->{$key} = $ips;
+        } elsif ($key eq 'MAX_WORKERS') {
+            if ($value =~ /^\d+$/) {
+                $res->{$key} = int($value);
+            } else {
+                warn "MAX_WORKERS specified in $conffile is not an integer: $value\n";
+            }
         } elsif (grep { $key eq $_ } @$boolean_options) {
             die "unknown value '$value' - use 0 or 1\n" if $value !~ m/^(0|1)$/;
             $res->{$key} = $value;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 1/2] partially fix #5392: pveproxy: make number of workers configurable

[Friedrich Weber]

The number of pveproxy worker processes is currently hardcoded to 3.
This may not be enough for automation-heavy workloads that trigger a
lot of API requests that are synchronously handled by pveproxy.

Hence, allow specifying MAX_WORKERS in /etc/default/pveproxy to
override the number of workers.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    I decided against setting max_workers directly directly in
    %daemon_options, to avoid having to call read_proxy_config already
    then. If I understand correctly, overriding $self->{max_workers} in
    init should be fine because it's only used in PVE::Daemon's
    $server_run after init was called.

 PVE/Service/pveproxy.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/PVE/Service/pveproxy.pm b/PVE/Service/pveproxy.pm
index c4bb54ea..ab455ff9 100755
--- a/PVE/Service/pveproxy.pm
+++ b/PVE/Service/pveproxy.pm
@@ -29,7 +29,7 @@ use base qw(PVE::Daemon);
 my $cmdline = [$0, @ARGV];
 
 my %daemon_options = (
-    max_workers => 3,
+    max_workers => 3, # may be overridden in init
     restart_on_error => 5,
     stop_wait_time => 15,
     leave_children_open_on_reload => 1,
@@ -64,6 +64,7 @@ sub init {
 
     # we use same ALLOW/DENY/POLICY as pveproxy
     my $proxyconf = PVE::APIServer::Utils::read_proxy_config($self->{name});
+    $self->{max_workers} = $proxyconf->{MAX_WORKERS} if $proxyconf->{MAX_WORKERS};
 
     my $accept_lock_fn = "/var/lock/pveproxy.lck";
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 2/2] partially fix #5392: pvedaemon: make number of workers configurable

[Friedrich Weber]

The number of pvedaemon worker processes is currently hardcoded to 3.
This may not be enough for automation-heavy workloads that trigger a
lot of API requests that are synchronously handled by pvedaemon.

Hence, read /etc/default/pvedaemon when starting pvedaemon and allow
overriding the number of workers by specifying MAX_WORKERS in this
file. All other values are only relevant for pveproxy/spiceproxy and
thus ignored.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    I'm not sure if reading /etc/default/pvedaemon, which was not used
    until now, is a good solution here, but I decided to go for it because
    it seems relatively straightforward and analogous to pveproxy.
    
    Also here, I decided against setting max_workers directly directly in
    %daemon_options, to avoid having to call read_proxy_config already
    then. If I understand correctly, overriding $self->{max_workers} in
    init should be fine because it's only used in PVE::Daemon's
    $server_run after init was called.

 PVE/Service/pvedaemon.pm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/PVE/Service/pvedaemon.pm b/PVE/Service/pvedaemon.pm
index 9d7cbc0f..4493f234 100755
--- a/PVE/Service/pvedaemon.pm
+++ b/PVE/Service/pvedaemon.pm
@@ -15,7 +15,7 @@ use base qw(PVE::Daemon);
 my $cmdline = [$0, @ARGV];
 
 my %daemon_options = (
-    max_workers => 3,
+    max_workers => 3, # may be overridden in init
     restart_on_error => 5,
     stop_wait_time => 15,
     leave_children_open_on_reload => 1,
@@ -26,6 +26,10 @@ my $daemon = __PACKAGE__->new('pvedaemon', $cmdline, %daemon_options);
 sub init {
     my ($self) = @_;
 
+    # all options other than MAX_WORKERS are ignored
+    my $proxyconf = PVE::APIServer::Utils::read_proxy_config($self->{name});
+    $self->{max_workers} = $proxyconf->{MAX_WORKERS} if $proxyconf->{MAX_WORKERS};
+
     my $accept_lock_fn = "/var/lock/pvedaemon.lck";
 
     my $lockfh = IO::File->new(">>${accept_lock_fn}")
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH http-server 1/1] api server: proxy config: read MAX_WORKERS integer key

[Thomas Lamprecht]

Am 29.07.25 um 13:35 schrieb Friedrich Weber:
> Read the MAX_WORKERS value in /etc/default/<proxyname>. If it is not
> an integer, ignore and warn.
> 
> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
> ---
>  src/PVE/APIServer/Utils.pm | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/src/PVE/APIServer/Utils.pm b/src/PVE/APIServer/Utils.pm
> index 1430c98..f2c4892 100644
> --- a/src/PVE/APIServer/Utils.pm
> +++ b/src/PVE/APIServer/Utils.pm
> @@ -28,6 +28,7 @@ sub read_proxy_config {
>      $shcmd .= 'echo \"DISABLE_TLS_1_3:\$DISABLE_TLS_1_3\";';
>      $shcmd .= 'echo \"PROXY_REAL_IP_HEADER:\$PROXY_REAL_IP_HEADER\";';
>      $shcmd .= 'echo \"PROXY_REAL_IP_ALLOW_FROM:\$PROXY_REAL_IP_ALLOW_FROM\";';
> +    $shcmd .= 'echo \"MAX_WORKERS:\$MAX_WORKERS\";';
>  
>      my $data = -f $conffile ? `bash -c "$shcmd"` : '';
>  
> @@ -77,6 +78,12 @@ sub read_proxy_config {
>                  push @$ips, Net::IP->new(normalize_v4_in_v6($ip)) || die Net::IP::Error() . "\n";
>              }
>              $res->{$key} = $ips;
> +        } elsif ($key eq 'MAX_WORKERS') {
> +            if ($value =~ /^\d+$/) {
> +                $res->{$key} = int($value);

Could be great to do some basic range checks, like > 0 and < 128 (as not low, but
also not huge upper limit for starters).

FWIW, rest looks pretty straight forward and really not much extra code, so would
be fine for me to do it this way.

> +            } else {
> +                warn "MAX_WORKERS specified in $conffile is not an integer: $value\n";
> +            }
>          } elsif (grep { $key eq $_ } @$boolean_options) {
>              die "unknown value '$value' - use 0 or 1\n" if $value !~ m/^(0|1)$/;
>              $res->{$key} = $value;



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH http-server 1/1] api server: proxy config: read MAX_WORKERS integer key

[Friedrich Weber]

On 29/07/2025 13:44, Thomas Lamprecht wrote:
> Am 29.07.25 um 13:35 schrieb Friedrich Weber:
>> Read the MAX_WORKERS value in /etc/default/<proxyname>. If it is not
>> an integer, ignore and warn.
>>
>> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
>> ---
>>  src/PVE/APIServer/Utils.pm | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/src/PVE/APIServer/Utils.pm b/src/PVE/APIServer/Utils.pm
>> index 1430c98..f2c4892 100644
>> --- a/src/PVE/APIServer/Utils.pm
>> +++ b/src/PVE/APIServer/Utils.pm
>> @@ -28,6 +28,7 @@ sub read_proxy_config {
>>      $shcmd .= 'echo \"DISABLE_TLS_1_3:\$DISABLE_TLS_1_3\";';
>>      $shcmd .= 'echo \"PROXY_REAL_IP_HEADER:\$PROXY_REAL_IP_HEADER\";';
>>      $shcmd .= 'echo \"PROXY_REAL_IP_ALLOW_FROM:\$PROXY_REAL_IP_ALLOW_FROM\";';
>> +    $shcmd .= 'echo \"MAX_WORKERS:\$MAX_WORKERS\";';
>>  
>>      my $data = -f $conffile ? `bash -c "$shcmd"` : '';
>>  
>> @@ -77,6 +78,12 @@ sub read_proxy_config {
>>                  push @$ips, Net::IP->new(normalize_v4_in_v6($ip)) || die Net::IP::Error() . "\n";
>>              }
>>              $res->{$key} = $ips;
>> +        } elsif ($key eq 'MAX_WORKERS') {
>> +            if ($value =~ /^\d+$/) {
>> +                $res->{$key} = int($value);
> 
> Could be great to do some basic range checks, like > 0 and < 128 (as not low, but
> also not huge upper limit for starters).

Good point, thanks, I added this in the v1:

https://lore.proxmox.com/pve-devel/20250729155227.157120-1-f.weber@proxmox.com/

> 
> FWIW, rest looks pretty straight forward and really not much extra code, so would
> be fine for me to do it this way.
> 
>> +            } else {
>> +                warn "MAX_WORKERS specified in $conffile is not an integer: $value\n";
>> +            }
>>          } elsif (grep { $key eq $_ } @$boolean_options) {
>>              die "unknown value '$value' - use 0 or 1\n" if $value !~ m/^(0|1)$/;
>>              $res->{$key} = $value;
> 



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] superseded: [RFC http-server/manager 0/3] fix #5392: pveproxy, pvedaemon: make number of worker processes configurable

[Friedrich Weber]

Superseded by:

https://lore.proxmox.com/pve-devel/20250729155227.157120-1-f.weber@proxmox.com/


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH http-server 1/1] api server: proxy config: read MAX_WORKERS integer key

[Friedrich Weber]

Read the MAX_WORKERS value in /etc/default/<proxyname>. If it is not
an integer between 0 and 128, ignore and warn.

The lower limit was chosen because at least one worker process must
exist. The upper limit was chosen because more than 127 worker
processes should not be necessary and a positive impact on performance
is doubtful. If this limit turns out to be too low, it can still be
extended in the future.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    changes since rfc:
    - add lower and upper limit (thx Thomas!)

 src/PVE/APIServer/Utils.pm | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/PVE/APIServer/Utils.pm b/src/PVE/APIServer/Utils.pm
index 1430c98..4cceb71 100644
--- a/src/PVE/APIServer/Utils.pm
+++ b/src/PVE/APIServer/Utils.pm
@@ -28,6 +28,7 @@ sub read_proxy_config {
     $shcmd .= 'echo \"DISABLE_TLS_1_3:\$DISABLE_TLS_1_3\";';
     $shcmd .= 'echo \"PROXY_REAL_IP_HEADER:\$PROXY_REAL_IP_HEADER\";';
     $shcmd .= 'echo \"PROXY_REAL_IP_ALLOW_FROM:\$PROXY_REAL_IP_ALLOW_FROM\";';
+    $shcmd .= 'echo \"MAX_WORKERS:\$MAX_WORKERS\";';
 
     my $data = -f $conffile ? `bash -c "$shcmd"` : '';
 
@@ -77,6 +78,13 @@ sub read_proxy_config {
                 push @$ips, Net::IP->new(normalize_v4_in_v6($ip)) || die Net::IP::Error() . "\n";
             }
             $res->{$key} = $ips;
+        } elsif ($key eq 'MAX_WORKERS') {
+            if ($value =~ /^\d+$/ && $value > 0 && $value < 128) {
+                $res->{$key} = int($value);
+            } else {
+                warn "MAX_WORKERS specified in $conffile is not an integer between"
+                    ." 0 and 128: $value\n";
+            }
         } elsif (grep { $key eq $_ } @$boolean_options) {
             die "unknown value '$value' - use 0 or 1\n" if $value !~ m/^(0|1)$/;
             $res->{$key} = $value;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel