From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id AAA321FF148 for ; Sun, 12 Apr 2026 13:12:08 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E85F7F8E1; Sun, 12 Apr 2026 13:12:53 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH http-server 0/1] fix pveproxy OOM during PDM cross-cluster migration to LVM-thin Date: Sun, 12 Apr 2026 19:12:08 +0800 Message-ID: <20260412111209.3960421-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1775992265064 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.372 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: PJQ4QNVI26PSVJCEIB2F3JI4QPZPWXIT X-Message-ID-Hash: PJQ4QNVI26PSVJCEIB2F3JI4QPZPWXIT X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: pveproxy on the destination host can be OOM-killed during PDM offline cross-cluster migration when the target storage (e.g. LVM-thin) writes more slowly than data arrives over the network [1]. the websocket proxy already had wbuf_max set on the backend handle, but it turns out not to help. AnyEvent::Handle only checks it inside the `if (!$self->{_ww})` guard in _drain_wbuf: if (!$self->{_ww} && length $self->{wbuf}) { ... if (defined $self->{wbuf_max} && $self->{wbuf_max} < ...) { $self->_error(Errno::ENOSPC, 1); } } once the first EAGAIN installs the write watcher (_ww), all subsequent push_write calls return immediately without reaching the check, so wbuf grows without bound. to fix this, follow the same approach response_stream() already takes: stop reading from the source handle when the backend write buffer exceeds the limit, and resume via on_drain once it empties. handle_spice_proxy_request() has the same issue and even carries a "# todo: use stop_read/start_read" comment acknowledging it, but is not addressed here as SPICE carries interactive VM console traffic rather than bulk data. tested with a synthetic AnyEvent script that drives a fast writer through a proxy into a slow reader. without backpressure, the proxy write buffer grows to ~1.4 GB in 5 seconds (2254x the limit). with the fix, it stays bounded at just over the 640 KB limit, as expected. [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7483 Kefu Chai (1): fix #7483: apiserver: add backpressure to websocket proxy src/PVE/APIServer/AnyEvent.pm | 45 +++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 7 deletions(-) -- 2.47.3