[pve-devel] [RFC PATCH http-server] fix #6230: increase allowed post size

[pve-devel] [RFC PATCH http-server] fix #6230: increase allowed post size

[Dominik Csapak]

In some situations, e.g. having a large resource mapping, the UI can
generate a request that is bigger than the current limit of 64KiB.

Our files in pmxcfs can grow up to 1 MiB, so theoretically, a single
mapping can grow to that size. In practice, a single entry will have
much less. In #6230, a user has a mapping with about ~130KiB.

Increase the limit to 512KiB so we have a bit of buffer left.

We have to also increase the 'rbuf_max' size here, otherwise the request
will fail (since the buffer is too small for the request).
Since the post limit and the rbuf_max are tightly coupled, let it
reflect that in the code. To do that sum the post size + max header
size there.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
sending as RFC because:
* not sure about the rbuf_max calculation, but we have to increase it
  when we increase $limit_max_post. (not sure how much is needed exactly)
* ther are alternative ways to deal with that, but some of those are vastly
  more work:
  - optimize the pci mapping to reduce the number of bytes we have to
    send (e.g. by reducing the property names, or somehow magically
    detect devices that belong together)
  - add a new api for the mappings that can update the entries without
    sending the whole mapping again (not sure if we can make this
    backwards compatible)
  - ignore the problem and simply tell the users to edit the file
    manually (I don't like this one...)

also, I tried to benchmark this, but did not find a tool that does this
in a good way (e.g. apachebench complained about ssl, and i couldn't get
it to work right). @Thomas you did such benchmarks laft according to git
log, do you remember what you used then?

 src/PVE/APIServer/AnyEvent.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
index 8a52836..43ced75 100644
--- a/src/PVE/APIServer/AnyEvent.pm
+++ b/src/PVE/APIServer/AnyEvent.pm
@@ -52,7 +52,7 @@ use PVE::APIServer::Utils;
 
 my $limit_max_headers = 64;
 my $limit_max_header_size = 8*1024;
-my $limit_max_post = 64*1024;
+my $limit_max_post = 512*1024;
 
 my $known_methods = {
     GET => 1,
@@ -1891,7 +1891,7 @@ sub accept_connections {
 	    $self->{conn_count}++;
 	    $reqstate->{hdl} = AnyEvent::Handle->new(
 		fh => $clientfh,
-		rbuf_max => 64*1024,
+		rbuf_max => $limit_max_post + ($limit_max_headers * $limit_max_header_size),
 		timeout => $self->{timeout},
 		linger => 0, # avoid problems with ssh - really needed ?
 		on_eof   => sub {
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC PATCH http-server] fix #6230: increase allowed post size

[Savely Krasovsky via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 8850 bytes --]

From: Savely Krasovsky <savely@krasovs.ky>
To: pve-devel@lists.proxmox.com
Subject: Re: [pve-devel] [RFC PATCH http-server] fix #6230: increase allowed post size
Date: Wed, 2 Apr 2025 21:45:46 +0200
Message-ID: <5f884d58-2703-4a0f-8859-d090847497b7@krasovs.ky>

In my case I provision VMs using Terraform and Ignition and the only way to do it it to set
-fw_cfg with giant inline JSON. At first it was fine, but now my config exceeded
~40Kb and I faced with this issue. This patch helped, hope this will be fixed.

12.03.2025 14:27, Dominik Csapak пишет:

> In some situations, e.g. having a large resource mapping, the UI can
> generate a request that is bigger than the current limit of 64KiB.
>
> Our files in pmxcfs can grow up to 1 MiB, so theoretically, a single
> mapping can grow to that size. In practice, a single entry will have
> much less. In #6230, a user has a mapping with about ~130KiB.
>
> Increase the limit to 512KiB so we have a bit of buffer left.
>
> We have to also increase the 'rbuf_max' size here, otherwise the request
> will fail (since the buffer is too small for the request).
> Since the post limit and the rbuf_max are tightly coupled, let it
> reflect that in the code. To do that sum the post size + max header
> size there.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> sending as RFC because:
> * not sure about the rbuf_max calculation, but we have to increase it
>    when we increase $limit_max_post. (not sure how much is needed exactly)
> * ther are alternative ways to deal with that, but some of those are vastly
>    more work:
>    - optimize the pci mapping to reduce the number of bytes we have to
>      send (e.g. by reducing the property names, or somehow magically
>      detect devices that belong together)
>    - add a new api for the mappings that can update the entries without
>      sending the whole mapping again (not sure if we can make this
>      backwards compatible)
>    - ignore the problem and simply tell the users to edit the file
>      manually (I don't like this one...)
>
> also, I tried to benchmark this, but did not find a tool that does this
> in a good way (e.g. apachebench complained about ssl, and i couldn't get
> it to work right). @Thomas you did such benchmarks laft according to git
> log, do you remember what you used then?
>
>   src/PVE/APIServer/AnyEvent.pm | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
> index 8a52836..43ced75 100644
> --- a/src/PVE/APIServer/AnyEvent.pm
> +++ b/src/PVE/APIServer/AnyEvent.pm
> @@ -52,7 +52,7 @@ use PVE::APIServer::Utils;
>   
>   my $limit_max_headers = 64;
>   my $limit_max_header_size = 8*1024;
> -my $limit_max_post = 64*1024;
> +my $limit_max_post = 512*1024;
>   
>   my $known_methods = {
>       GET => 1,
> @@ -1891,7 +1891,7 @@ sub accept_connections {
>   	    $self->{conn_count}++;
>   	    $reqstate->{hdl} = AnyEvent::Handle->new(
>   		fh => $clientfh,
> -		rbuf_max => 64*1024,
> +		rbuf_max => $limit_max_post + ($limit_max_headers * $limit_max_header_size),
>   		timeout => $self->{timeout},
>   		linger => 0, # avoid problems with ssh - really needed ?
>   		on_eof   => sub {



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC PATCH http-server] fix #6230: increase allowed post size

[Thomas Lamprecht]

Am 12.03.25 um 14:27 schrieb Dominik Csapak:
> In some situations, e.g. having a large resource mapping, the UI can
> generate a request that is bigger than the current limit of 64KiB.
> 
> Our files in pmxcfs can grow up to 1 MiB, so theoretically, a single
> mapping can grow to that size. In practice, a single entry will have
> much less. In #6230, a user has a mapping with about ~130KiB.
> 
> Increase the limit to 512KiB so we have a bit of buffer left.

s/buffer/headroom/ ?

> 
> We have to also increase the 'rbuf_max' size here, otherwise the request
> will fail (since the buffer is too small for the request).
> Since the post limit and the rbuf_max are tightly coupled, let it
> reflect that in the code. To do that sum the post size + max header
> size there.
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> sending as RFC because:
> * not sure about the rbuf_max calculation, but we have to increase it
>   when we increase $limit_max_post. (not sure how much is needed exactly)
> * ther are alternative ways to deal with that, but some of those are vastly
>   more work:
>   - optimize the pci mapping to reduce the number of bytes we have to
>     send (e.g. by reducing the property names, or somehow magically
>     detect devices that belong together)
>   - add a new api for the mappings that can update the entries without
>     sending the whole mapping again (not sure if we can make this
>     backwards compatible)
>   - ignore the problem and simply tell the users to edit the file
>     manually (I don't like this one...)
> 
> also, I tried to benchmark this, but did not find a tool that does this
> in a good way (e.g. apachebench complained about ssl, and i couldn't get
> it to work right). @Thomas you did such benchmarks laft according to git
> log, do you remember what you used then?

argh, my commit message back then looks like I tried to write what I used
but then fubmled (or got knocked on the head) and sent it out unfinished.
To my defence, Wolfgang applied it ;P

I'm not totally sure what I used back then, might have been something
custom-made too. FWIW, recently I used oha [0] and found it quite OK, albeit
I did not try it with POST data, but one can define the method and pass a
request body from CLI argument directly or a file, and it has a flag to
allow "insecure" TLS certs.

[0]: https://github.com/hatoo/oha

> @@ -1891,7 +1891,7 @@ sub accept_connections {
>  	    $self->{conn_count}++;
>  	    $reqstate->{hdl} = AnyEvent::Handle->new(
>  		fh => $clientfh,
> -		rbuf_max => 64*1024,
> +		rbuf_max => $limit_max_post + ($limit_max_headers * $limit_max_header_size),

The header part is wrong as the header limits are independent, i.e., the
request must have less than $limit_max_headers separate headers and all
those together must be smaller than $limit_max_header_size.

So just adding $limit_max_header_size is enough, no multiplication required.

>  		timeout => $self->{timeout},
>  		linger => 0, # avoid problems with ssh - really needed ?
>  		on_eof   => sub {


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC PATCH http-server] fix #6230: increase allowed post size

[Dominik Csapak]

On 4/2/25 22:09, Thomas Lamprecht wrote:
> Am 12.03.25 um 14:27 schrieb Dominik Csapak:
>> In some situations, e.g. having a large resource mapping, the UI can
>> generate a request that is bigger than the current limit of 64KiB.
>>
>> Our files in pmxcfs can grow up to 1 MiB, so theoretically, a single
>> mapping can grow to that size. In practice, a single entry will have
>> much less. In #6230, a user has a mapping with about ~130KiB.
>>
>> Increase the limit to 512KiB so we have a bit of buffer left.
> 
> s/buffer/headroom/ ?
> 

yes, makes more sense^^

>>
>> We have to also increase the 'rbuf_max' size here, otherwise the request
>> will fail (since the buffer is too small for the request).
>> Since the post limit and the rbuf_max are tightly coupled, let it
>> reflect that in the code. To do that sum the post size + max header
>> size there.
>>
>> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
>> ---
>> sending as RFC because:
>> * not sure about the rbuf_max calculation, but we have to increase it
>>    when we increase $limit_max_post. (not sure how much is needed exactly)
>> * ther are alternative ways to deal with that, but some of those are vastly
>>    more work:
>>    - optimize the pci mapping to reduce the number of bytes we have to
>>      send (e.g. by reducing the property names, or somehow magically
>>      detect devices that belong together)
>>    - add a new api for the mappings that can update the entries without
>>      sending the whole mapping again (not sure if we can make this
>>      backwards compatible)
>>    - ignore the problem and simply tell the users to edit the file
>>      manually (I don't like this one...)
>>
>> also, I tried to benchmark this, but did not find a tool that does this
>> in a good way (e.g. apachebench complained about ssl, and i couldn't get
>> it to work right). @Thomas you did such benchmarks laft according to git
>> log, do you remember what you used then?
> 
> argh, my commit message back then looks like I tried to write what I used
> but then fubmled (or got knocked on the head) and sent it out unfinished.
> To my defence, Wolfgang applied it ;P
> 
> I'm not totally sure what I used back then, might have been something
> custom-made too. FWIW, recently I used oha [0] and found it quite OK, albeit
> I did not try it with POST data, but one can define the method and pass a
> request body from CLI argument directly or a file, and it has a flag to
> allow "insecure" TLS certs.
> 
> [0]: https://github.com/hatoo/oha

thanks, i'll try to do some benchmarks with it

> 
>> @@ -1891,7 +1891,7 @@ sub accept_connections {
>>   	    $self->{conn_count}++;
>>   	    $reqstate->{hdl} = AnyEvent::Handle->new(
>>   		fh => $clientfh,
>> -		rbuf_max => 64*1024,
>> +		rbuf_max => $limit_max_post + ($limit_max_headers * $limit_max_header_size),
> 
> The header part is wrong as the header limits are independent, i.e., the
> request must have less than $limit_max_headers separate headers and all
> those together must be smaller than $limit_max_header_size.
> 
> So just adding $limit_max_header_size is enough, no multiplication required.
> 

ah yes, seems i read those wrong

>>   		timeout => $self->{timeout},
>>   		linger => 0, # avoid problems with ssh - really needed ?
>>   		on_eof   => sub {



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC PATCH http-server] fix #6230: increase allowed post size

[Dominik Csapak]

sent a new version:
https://lore.proxmox.com/pve-devel/20250403082759.2506153-1-d.csapak@proxmox.com/


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel