From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id A71531FF17C
for <inbox@lore.proxmox.com>; Wed, 2 Apr 2025 21:52:07 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 7FEDF31A1E;
Wed, 2 Apr 2025 21:51:56 +0200 (CEST)
Date: Wed, 2 Apr 2025 21:45:46 +0200
To: pve-devel@lists.proxmox.com
References: <20250312132738.2268305-1-d.csapak@proxmox.com>
In-Reply-To: <20250312132738.2268305-1-d.csapak@proxmox.com>
MIME-Version: 1.0
Message-ID: <mailman.533.1743623515.359.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Savely Krasovsky via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Savely Krasovsky <savely@krasovs.ky>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: Re: [pve-devel] [RFC PATCH http-server] fix #6230: increase allowed
post size
Content-Type: multipart/mixed; boundary="===============4221603521250261880=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
--===============4221603521250261880==
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <savely@krasovs.ky>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id BEBFECED11
for <pve-devel@lists.proxmox.com>; Wed, 2 Apr 2025 21:51:54 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id A8D3031996
for <pve-devel@lists.proxmox.com>; Wed, 2 Apr 2025 21:51:54 +0200 (CEST)
Received: from fout-b2-smtp.messagingengine.com (fout-b2-smtp.messagingengine.com [202.12.124.145])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Wed, 2 Apr 2025 21:51:53 +0200 (CEST)
Received: from phl-compute-11.internal (phl-compute-11.phl.internal [10.202.2.51])
by mailfout.stl.internal (Postfix) with ESMTP id 08A3E1140146
for <pve-devel@lists.proxmox.com>; Wed, 2 Apr 2025 15:45:49 -0400 (EDT)
Received: from phl-mailfrontend-02 ([10.202.2.163])
by phl-compute-11.internal (MEProxy); Wed, 02 Apr 2025 15:45:49 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=krasovs.ky; h=cc
:content-transfer-encoding:content-type:content-type:date:date
:from:from:in-reply-to:in-reply-to:message-id:mime-version
:references:reply-to:subject:subject:to:to; s=fm1; t=1743623148;
x=1743709548; bh=2ardEhNWL/xF4fW9ASV60Sb5IWDJnmXLEIUsPG838wQ=; b=
Kg27bd/+FAT/PqxKEHd6ZT3YMg81PM8Q35Wziw3WH5+WVlt6H2PoNV+RriANAIwq
Y6gJ8wJMKtoGSlVTZU6YFP0NtkwTBrWMWzcrMqkFHmbbj07l1m52Nn/ak3DIEEJ9
gYbOiWGSmYB/s2sVWKvOIg2ElGGwStzs139s+Ak3GVpVoWebs+1JJSef8f3z7o+T
qrNk0TDL/voS/sOz3aqud4Fnyqw+UYAf+HcDBZbyW3SzV6MuSVaIzsIECPnb5oud
A/SzP1Cv5rASY3pUqZvP8MiKlWUMWGMFyNpO/+aU8Aoi6fPZDXOFF0U7eRPZBn17
cDoWLtS2iSJ64WTGU8mUDg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=cc:content-transfer-encoding:content-type
:content-type:date:date:feedback-id:feedback-id:from:from
:in-reply-to:in-reply-to:message-id:mime-version:references
:reply-to:subject:subject:to:to:x-me-proxy:x-me-sender
:x-me-sender:x-sasl-enc; s=fm2; t=1743623148; x=1743709548; bh=2
ardEhNWL/xF4fW9ASV60Sb5IWDJnmXLEIUsPG838wQ=; b=aOVI1uFR19Ftre+8x
iNsLr80bRI7P/1baf+uVLCu5FqNQoC6HdibWl1koCyNnC40kJG8gq03YW8pK8+SK
wRsElAKvvGx4vZ+bnGk8fFZbO5i4vDM8aYr60suGJZ1BCN+r63xrnoeXNTSx+VsC
5Pz8mos/f8bEiWSGWo6QJ62vi9azcEaVdWGoiQYbtdpIedV9XXVXZCElf4mI3j2N
gu2sZwW0P5T2LwJxRjivfhM6Jo6rLrtXiIdQN9yZ9ZYWaylqXmVdQYqmp/F0WLpk
T5dW+tQD9dctpGQlp6qU0jWnUVtUs0omonvJtEZmLvx/2yQmHXdvN1cGXPYTy8/h
pM6Dw==
X-ME-Sender: <xms:7JPtZyaZrBeRaNACni7qrSnq027y932yGqQcNNOTDJoVWe1giIFmpw>
<xme:7JPtZ1Z1E5F2DOKjJ1Y5dNVAeQaBCaoyOlGe6psdH-AT9ZUTZIp-9s8gKwbHo9z7_
gJoRm077vZXuOWoORc>
X-ME-Received: <xmr:7JPtZ8-RXPcAfNEvkNo-RgO4gatL6p_zpEraQC21FvjDpaE2wcN9EdceoQYmv9ccdQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddukeeiheefucetufdoteggodetrf
dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv
pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpefkff
ggfgfuvfhfhfgjtgfgsehtkeertddtvdejnecuhfhrohhmpefurghvvghlhicumfhrrghs
ohhvshhkhicuoehsrghvvghlhieskhhrrghsohhvshdrkhihqeenucggtffrrghtthgvrh
hnpeekuefgvedvheekjeehheelhffgleevhffgudekieehudeuffekgeevffekieeufeen
ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrghvvg
hlhieskhhrrghsohhvshdrkhihpdhnsggprhgtphhtthhopedupdhmohguvgepshhmthhp
ohhuthdprhgtphhtthhopehpvhgvqdguvghvvghlsehlihhsthhsrdhprhhogihmohigrd
gtohhm
X-ME-Proxy: <xmx:7JPtZ0p7Ia9KLrP2Tdn9cuVnkptfOa5kpDbMBsVVM0MLdOrQ2fjGnw>
<xmx:7JPtZ9q-gbZ--Qh7HTpJZzzhasxYTNyf_sXn_BFd0FBykHtuiDpmmQ>
<xmx:7JPtZyS7lMSS4VAM154XRsIdLgU1-t-mGrhPQnlM8bA42MJHz53MBg>
<xmx:7JPtZ9qXQ4Pta2velZxzIUhXS_khxn0njykCGUaEi_Zn45YcEb3xoQ>
<xmx:7JPtZxIBPVqH7ki89bbRfbe9N1gB1PHbakSU3HOL8f95Cmhf_ALAGL-G>
Feedback-ID: ia7b94449:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA for
<pve-devel@lists.proxmox.com>; Wed, 2 Apr 2025 15:45:47 -0400 (EDT)
Message-ID: <5f884d58-2703-4a0f-8859-d090847497b7@krasovs.ky>
Date: Wed, 2 Apr 2025 21:45:46 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [pve-devel] [RFC PATCH http-server] fix #6230: increase allowed
post size
To: pve-devel@lists.proxmox.com
References: <20250312132738.2268305-1-d.csapak@proxmox.com>
Content-Language: ru
From: Savely Krasovsky <savely@krasovs.ky>
In-Reply-To: <20250312132738.2268305-1-d.csapak@proxmox.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid
DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain
DMARC_PASS -0.1 DMARC pass policy
JMQ_SPF_NEUTRAL 0.5 SPF set to ?all
RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust
SPF_HELO_PASS -0.001 SPF: HELO matches SPF record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [anyevent.pm,krasovs.ky,messagingengine.com]
In my case I provision VMs using Terraform and Ignition and the only way to do it it to set
-fw_cfg with giant inline JSON. At first it was fine, but now my config exceeded
~40Kb and I faced with this issue. This patch helped, hope this will be fixed.
12.03.2025 14:27, Dominik Csapak пишет:
> In some situations, e.g. having a large resource mapping, the UI can
> generate a request that is bigger than the current limit of 64KiB.
>
> Our files in pmxcfs can grow up to 1 MiB, so theoretically, a single
> mapping can grow to that size. In practice, a single entry will have
> much less. In #6230, a user has a mapping with about ~130KiB.
>
> Increase the limit to 512KiB so we have a bit of buffer left.
>
> We have to also increase the 'rbuf_max' size here, otherwise the request
> will fail (since the buffer is too small for the request).
> Since the post limit and the rbuf_max are tightly coupled, let it
> reflect that in the code. To do that sum the post size + max header
> size there.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> sending as RFC because:
> * not sure about the rbuf_max calculation, but we have to increase it
> when we increase $limit_max_post. (not sure how much is needed exactly)
> * ther are alternative ways to deal with that, but some of those are vastly
> more work:
> - optimize the pci mapping to reduce the number of bytes we have to
> send (e.g. by reducing the property names, or somehow magically
> detect devices that belong together)
> - add a new api for the mappings that can update the entries without
> sending the whole mapping again (not sure if we can make this
> backwards compatible)
> - ignore the problem and simply tell the users to edit the file
> manually (I don't like this one...)
>
> also, I tried to benchmark this, but did not find a tool that does this
> in a good way (e.g. apachebench complained about ssl, and i couldn't get
> it to work right). @Thomas you did such benchmarks laft according to git
> log, do you remember what you used then?
>
> src/PVE/APIServer/AnyEvent.pm | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
> index 8a52836..43ced75 100644
> --- a/src/PVE/APIServer/AnyEvent.pm
> +++ b/src/PVE/APIServer/AnyEvent.pm
> @@ -52,7 +52,7 @@ use PVE::APIServer::Utils;
>
> my $limit_max_headers = 64;
> my $limit_max_header_size = 8*1024;
> -my $limit_max_post = 64*1024;
> +my $limit_max_post = 512*1024;
>
> my $known_methods = {
> GET => 1,
> @@ -1891,7 +1891,7 @@ sub accept_connections {
> $self->{conn_count}++;
> $reqstate->{hdl} = AnyEvent::Handle->new(
> fh => $clientfh,
> - rbuf_max => 64*1024,
> + rbuf_max => $limit_max_post + ($limit_max_headers * $limit_max_header_size),
> timeout => $self->{timeout},
> linger => 0, # avoid problems with ssh - really needed ?
> on_eof => sub {
--===============4221603521250261880==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--===============4221603521250261880==--