all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Kefu Chai <k.chai@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH http-server 0/1] fix pveproxy OOM during PDM cross-cluster migration to LVM-thin
Date: Sun, 12 Apr 2026 19:12:08 +0800	[thread overview]
Message-ID: <20260412111209.3960421-1-k.chai@proxmox.com> (raw)

pveproxy on the destination host can be OOM-killed during PDM offline
cross-cluster migration when the target storage (e.g. LVM-thin) writes more
slowly than data arrives over the network [1].

the websocket proxy already had wbuf_max set on the backend handle, but it
turns out not to help. AnyEvent::Handle only checks it inside the
`if (!$self->{_ww})` guard in _drain_wbuf:

    if (!$self->{_ww} && length $self->{wbuf}) {
        ...
        if (defined $self->{wbuf_max} && $self->{wbuf_max} < ...) {
            $self->_error(Errno::ENOSPC, 1);
        }
    }

once the first EAGAIN installs the write watcher (_ww), all subsequent
push_write calls return immediately without reaching the check, so wbuf
grows without bound.

to fix this, follow the same approach response_stream() already takes: stop
reading from the source handle when the backend write buffer exceeds the
limit, and resume via on_drain once it empties.

handle_spice_proxy_request() has the same issue and even carries a
"# todo: use stop_read/start_read" comment acknowledging it, but is not
addressed here as SPICE carries interactive VM console traffic rather than
bulk data.

tested with a synthetic AnyEvent script that drives a fast writer through a
proxy into a slow reader. without backpressure, the proxy write buffer grows
to ~1.4 GB in 5 seconds (2254x the limit). with the fix, it stays bounded at
just over the 640 KB limit, as expected.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7483

Kefu Chai (1):
  fix #7483: apiserver: add backpressure to websocket proxy

 src/PVE/APIServer/AnyEvent.pm | 45 +++++++++++++++++++++++++++++------
 1 file changed, 38 insertions(+), 7 deletions(-)

-- 
2.47.3





             reply	other threads:[~2026-04-12 11:12 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-04-12 11:12 Kefu Chai [this message]
2026-04-12 11:12 ` [PATCH http-server 1/1] fix #7483: apiserver: add backpressure to websocket proxy Kefu Chai

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260412111209.3960421-1-k.chai@proxmox.com \
    --to=k.chai@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal