From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 277821FF0B7 for ; Tue, 25 Aug 2026 13:35:18 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 20F5321611; Tue, 25 Aug 2026 13:35:13 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tSJoBZfbyBAXYLHiJxfrkF+6f0olblT2ZCQkzaa/nSOLviH0UKQqdaKM0/KRlRno5iDEXX16Eamc37iyWbInnv7iFCqu0ld4Ej2Z7ziYJ6f7TCpgkcFfFMqr5c5wJgr807P15zhFIlioJSOW9eDuSNWi/YC0HRIFcINbl9locvuUJXRNVKxfoW/FyW51+38rQaLRfI5nc627RwCNqbcZexzvkR+NDmq/ZdGbj2B7GXtw3RCLRGPwOSiEBQPmZelap5EgAWZiv1XUQTFDff+LtGFiMg25ckURObbvQ/m8eh/vURTdSLaVmKkaHV7ahmXKc3vVYBDRW0SN2sDfeMLXNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=C79qd4DNCoLc4eckJFHKm7IGZQm70oKLZco/HVdZpj0=; b=enJT3hyrgSuXS3nHrXXCl9UdeHbytnWmU7Lu1YnpbZnqCbJQnG1A7WonUnebATDHR57G+KZG3D4XIZBtz7aC+V5C4l1eQpJoNGf/Zq1D8nDkY07QBKj5G1TT1hquk947RqTF5j5Rhc6sR6F1rpLz71WT9akqXnDTS0S8LUkAZI39oasGpB6TxszaBKskzxbBKdkMSJKmWS4oL0phwXa7/l6dgK7bL1HGGsgFc63RaGeWw2e1DzKrJ8bLaKTfUJS0tqv44y2naw9kO0FycYKJFz+dhYLGPMNq/ylCUDSlowdUswUClpLJZxcarYh2J1soh+VwJCd9BpOSOxUTN0hI2Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=groupe-cyllene.com; dmarc=pass action=none header.from=groupe-cyllene.com; dkim=pass header.d=groupe-cyllene.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=groupecyllene.onmicrosoft.com; s=selector2-groupecyllene-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C79qd4DNCoLc4eckJFHKm7IGZQm70oKLZco/HVdZpj0=; b=elu1VzRSM7wWKCoYR6kP45i0R9Qs82yTcmPUAyZSohbvLiATpQY+iwLTZUu2WGokNIpwE9C5uYKuBXfiVb1nVQiOcNj+5NwK9dEioOV++EQfIbkg3y2Ay2+5heuH+XZHr9ntdSXc3LMys3jqFXNqe8Tm67qc+PpQ3RTPqOewG9w= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=groupe-cyllene.com; From: Alexandre Derumier To: pve-devel@lists.proxmox.com Subject: [RFC pve-http-server 01/13] anyevent : proxy a path prefix to a local http proxy Date: Tue, 25 Aug 2026 13:34:26 +0200 Message-ID: <20260825113442.947620-2-alexandre.derumier@groupe-cyllene.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260825113442.947620-1-alexandre.derumier@groupe-cyllene.com> References: <20260825113442.947620-1-alexandre.derumier@groupe-cyllene.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: PR3P192CA0021.EURP192.PROD.OUTLOOK.COM (2603:10a6:102:56::26) To PR0P264MB3691.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:14b::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PR0P264MB3691:EE_|MRYP264MB6064:EE_ X-MS-Office365-Filtering-Correlation-Id: 0732b332-d23b-444a-9ad0-08df029ce407 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|23010399003|366016|1800799024|38350700014|6133799003|10067099003|56012099006|5023799004|18002099003|22082099003|3023799007; X-Microsoft-Antispam-Message-Info: 8A7gVRL67MKQTmzqXt9rJ97C3qjsQDHJoFU+PfOaccc3rytVh9l3r5T1pt+7ZUmOsPdAJWMGhOngmypgUYv16vd2nPqyuQQH4KizPEfUrre8UiDM2e4aaBghp6ZkeoEUgMEAtTqEO8Yok3HSiI8ZbyWLzEyneIsPKaASg5FBJ7bmCZ3j9ZhLsSxpEk1o7yxgVFcLHIh1/xGnuygXMWvALSghauDMCzk6mRlUmmg4mjaLIB71sFw7w0ygWkykOIaZxUlLFaoE0hCHGV4nDzfD39wU5+NOcUX42/EptBB4bCXTTiFNscsqYO8hxe61aeJii732U3JsAE+ZYUxNgsip2LF5Q1AcyUvy7TEXf0ev4aqC85ejSkEmG8ODB6+JnM/VkbiU2G8auN1tl+VA/Z52vSBW1bnrHRw86ZIwd4hrPoRRj0Bd1abVuP79AMql26y/KBWLI8KlKJilvkdjld3BJ/ZpfKwQb2XrTM5G/th0kQgrGlYCvem/ReLo1epa5u0tEkVqu3T79FnFcRardy72N8IvZ1z2w4Rb8VL9d8t/gngm+QY4JL7MlGiS6DNjD9k8O4ToXMg3MtttaYbgi1GJglsGtD2MyRs9ftcgh6EGUjUnDGofHhycxe7Vm6n+cSonJv1PTahbQBipYc52WhqG3ADpE95W8Cbhq1aTtAEcKRSdkMgTv0Rnu1nug8gT9W7IFdwINyHXWkX8CN7pjgDW89g66fytiiebj/i2sCtBRLE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PR0P264MB3691.FRAP264.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(23010399003)(366016)(1800799024)(38350700014)(6133799003)(10067099003)(56012099006)(5023799004)(18002099003)(22082099003)(3023799007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1hZWPgpzGRUb7g+4/IclONBH04VSDtWrz7+NdHJcIJKuNxy26oUVUkbyUygy?= =?us-ascii?Q?GrSiDSH1aY4koSGiu07srGMX1ppvJpgyggA7TIfIsGCHcU4tkikwaWVKTbRJ?= =?us-ascii?Q?THJ7X6tkNEXPR6WpyPk53ZIx3j0NL8MK3nE9i2lW9VZAGlf/ErqiDQ7C0hvP?= =?us-ascii?Q?iw3Tht9ZfwIzs69w7rFzlA283TuA7949d4E8PMv6rKliUB22fOUP8WzyOUaK?= =?us-ascii?Q?mJu4IHItkSHDubD+Iq8JHiggp04SGkd4TNEIigXDEYXRfGEiUxkM6Iog0DsM?= =?us-ascii?Q?McmOJyirGTVRGTBu7IfggiDHeJm+GJz0PngCgwZxroRtgXQTqEh6dWEuwzZP?= =?us-ascii?Q?MkjnZmQThN4OGaeSGrWkSwQl7Rd4EimPq/VkMf5XuJ9D8pzpZQxhZLxopAeJ?= =?us-ascii?Q?HqoJkYPqrdijVNfq7EUlY3q93pWKNwQ4Bk9FZUfItk05iXh2uXE8P6exFkAT?= =?us-ascii?Q?NTMGCtnNY0mWdatvMZz19oNghxCwqZ5EZoQfwVxGOCw5sNuJxIrBPy7Rs+8v?= =?us-ascii?Q?idIr/+omSv0DSnoinsdLAdULwhTQvROcGbjhNHP5xUUdpuPlZzdAzLKguc1I?= =?us-ascii?Q?uWkS0Gx7zCAAmiMQuywASX3h5XDsDy911ZM/MVcHuZfsANQGcqZ5GETKziSk?= =?us-ascii?Q?TjLjmbPLcSp7W/I0kLCFVlgJnAvMaLHGiuDW2+ef2AIRQfxf1Df9lhn7csIU?= =?us-ascii?Q?+ad6RkaHPxbWTdTsby1IOoPVlssHA2FLTgrKMRGQBbvNNmxpIA4dieOcStdH?= =?us-ascii?Q?q7buEFFkKo2FjeGfSuD0rMRf7FXulg1koKksVRaoXKDr43ZI0UX4nyuAbQfW?= =?us-ascii?Q?75tD7CyZRr1IOPXF+r1q5ZnpEhkjbc9vM48w+bUcwYKRV4ZDZe2nRJ00L5Hw?= =?us-ascii?Q?XGWrKnoewLrgD4o7NEe4nPB2LWJ021OcMXxiUUxTexhwcFMRoxbJa1At8IlW?= =?us-ascii?Q?jsHs3e2nq9JRTX0gYitZJp3xkFFvMrAe+2Q3WZKCCY0Koyt4NFrUuMLdSTNK?= =?us-ascii?Q?4JLAyzVRrrBtflZpsZf6KMXj6W3nseEAtHaRQvW+5yPVrNdDroEeNKYXNwZ6?= =?us-ascii?Q?T2kEEeHNo2GGmyt/fVZhME8Gmt9TkSHEzabWiH1lruLlwSwV5OslOH8epzVJ?= =?us-ascii?Q?VMHuqfeeI3vioEk/7jzSdG7vuWn9ee+UxVaDBvICg6w1Khxp2BH8z/+lxtgX?= =?us-ascii?Q?EZSCjBAd8Gfo7c3qBTPZdFRKK1gP46kkxF8vEqStf0EwX3IO2qgzIku3xyV6?= =?us-ascii?Q?tLLNNiwPcz3VR0ixqYNL3TOktxDrIXvHj+8eOb6T9o2Ohzsr2H8ES6OFmxfS?= =?us-ascii?Q?HWUIaOqleob9Zb+tLJf9RXc5qDxAls5RTHZuZtpSK1ezwclbRZm+kcnXkbMi?= =?us-ascii?Q?5gr3KCnZsC3PGzhroMcBv8gORtCxp06YTpZM7WH6by+PD4pYmyDP1cuoTg5J?= =?us-ascii?Q?haNfZCCtYKfX+gOasqnLO6Kw5BMw7yACBt1hKbOV3Pb/sRvtd7RNILZgI10R?= =?us-ascii?Q?gfASrHG+NvhgL3MFd6b1aRdTLdIwgHXemJRsK1nJkXF2So6Nh0i047GWYnHi?= =?us-ascii?Q?BaHu48G1/voe/Mb4Lb+EKiPx7v04J82DcLkWx5bEET2mTYdD3sQ+fWtfVHz7?= =?us-ascii?Q?nkAzgfbfDN+iLwnT25vwFAqNF8KE8uG80p5d8eWB7d3hT4xHsC5qzItz7c/9?= =?us-ascii?Q?L1KorIwE5+bw7x98JWl6Ev844uwgtS8H9BIot8WBIzMZgZNwem3cwaxYkQIX?= =?us-ascii?Q?2exVxnOxZtOkLlnp89vFdd4qG3qFGsjv7tk6ZD8Fs76IvLA2QuId?= X-OriginatorOrg: groupe-cyllene.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0732b332-d23b-444a-9ad0-08df029ce407 X-MS-Exchange-CrossTenant-AuthSource: PR0P264MB3691.FRAP264.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2026 11:34:57.6541 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: ee11ccf7-112c-4284-848b-f229745e715b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jemIYhpMusgB25oSDvtawt/b1NNDbrr5QUkSPX82/2JewFQvjDMcd6FUIG4nWyzzhBWK7IS3CNwPB2mf5joaEIYrpVeb62WUnO0nW6lL0Bm7FrC2shzJDlYNFHFsYTFv X-MS-Exchange-Transport-CrossTenantHeadersStamped: MRYP264MB6064 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.000 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: UHXKVFDDOS4RCDAPK4OLK3PUT3TA2CIJ X-Message-ID-Hash: UHXKVFDDOS4RCDAPK4OLK3PUT3TA2CIJ X-MailFrom: Alexandre.DERUMIER@groupe-cyllene.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A handler says which prefixes go to a local backend and who may reach them, so a service can use this server's TLS and authentication without its own port. Requests go verbatim, not re-encoded like proxy_request; an upgrade becomes a pipe after 101, and the relay stops reading while the far side is behind. Signed-off-by: Alexandre Derumier --- src/PVE/APIServer/AnyEvent.pm | 263 ++++++++++++++++++++++++++++++++++ 1 file changed, 263 insertions(+) diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm index 915d678..dc95c12 100644 --- a/src/PVE/APIServer/AnyEvent.pm +++ b/src/PVE/APIServer/AnyEvent.pm @@ -731,6 +731,250 @@ sub websocket_proxy { } } +# Queued for one side before the other stops being read. Smaller than +# response_stream's 4MB: consoles are not downloads, and a backlog is only +# latency the far end could have coalesced away. +my $relay_buf_size = 1024 * 1024; + +# What a handle still owes its socket; TLS keeps a second buffer. +sub relay_queued { + my ($hdl) = @_; + return length($hdl->{wbuf}) + length($hdl->{_tls_wbuf} // ''); +} + +# One direction of an upgraded connection: copy to the other side, and stop +# reading while that side is behind, so back pressure reaches the far end +# instead of queueing here. Same shape as response_stream, and named rather +# than a closure over itself, which would be a cycle. The handles come from +# callbacks because either may be gone by the time this runs. +sub relay_reader { + my ($from, $to) = @_; + + return sub { + my ($hdl) = @_; + + my $writer = $to->(); + return if !$writer; + + my $data = $hdl->{rbuf}; + $hdl->{rbuf} = ''; + $writer->push_write($data) if length($data); + + return if relay_queued($writer) < $relay_buf_size; + + my $prev_on_drain = $writer->{on_drain}; + $writer->on_drain(sub { + my ($wrhdl) = @_; + # Restored first: setting on_drain runs it on an empty buffer. + $wrhdl->on_drain($prev_on_drain); + if (my $reader = $from->()) { + $reader->on_read(relay_reader($from, $to)); + } + }); + + $hdl->on_read(); + }; +} + +# Hand an upgrade to the backend: the request goes out as it arrived and the +# answer comes back untouched, so the two ends compute the accept key. After +# 101 this is a pipe, which knows nothing of websockets. +sub local_http_proxy_upgrade { + my ($self, $reqstate, $method, $target) = @_; + + my $r = $reqstate->{request}; + + my ($remhost, $remport); + if ($target->{port}) { + $remhost = 'localhost'; + $remport = $target->{port}; + } else { + $remhost = 'unix/'; + $remport = $target->{socket}; + } + my $path = $target->{path} // '/'; + + # Only Host is rewritten: this is the hop being upgraded, so Connection + # and Upgrade stay. + my $headers = ''; + $r->headers->scan(sub { + my ($key, $value) = @_; + return if lc($key) eq 'host'; + $headers .= "$key: $value\015\012"; + }); + my $request = "$method $path HTTP/1.1\015\012Host: localhost\015\012$headers\015\012"; + + tcp_connect $remhost, $remport, sub { + my ($fh) = @_ + or do { + $self->error($reqstate, HTTP_BAD_GATEWAY, "connect to backend failed: $!"); + return; + }; + + $reqstate->{proxyhdl} = AnyEvent::Handle->new( + fh => $fh, + rbuf_max => 64 * 1024, + wbuf_max => 4 * $relay_buf_size, + timeout => 30, + on_eof => sub { + eval { + $self->log_aborted_request($reqstate); + $self->client_do_disconnect($reqstate); + }; + warn $@ if $@; + }, + on_error => sub { + my ($hdl, $fatal, $message) = @_; + eval { + $self->log_aborted_request($reqstate, $message); + $self->client_do_disconnect($reqstate); + }; + warn $@ if $@; + }, + ); + + $reqstate->{proxyhdl}->push_write($request); + + $reqstate->{proxyhdl}->push_read( + line => "\015\012\015\012", + sub { + my ($hdl, $response) = @_; + + # Only 101 means the backend stopped speaking HTTP. + if ($response !~ m|^HTTP/1\.1 101|) { + my ($status) = $response =~ m|^(\S+ \d+[^\015]*)|; + $self->log_aborted_request($reqstate, + "backend refused upgrade: " . ($status // 'unparseable response')); + $self->client_do_disconnect($reqstate); + return; + } + + # Verbatim: it carries the accept key for the client's key. + $reqstate->{hdl}->push_write($response . "\015\012\015\012"); + + $reqstate->{proxyhdl}->timeout(0); + $reqstate->{hdl}->timeout(0); + + my $client = sub { $reqstate->{hdl} }; + my $backend = sub { $reqstate->{proxyhdl} }; + + $reqstate->{proxyhdl}->on_read(relay_reader($backend, $client)); + $reqstate->{hdl}->on_read(relay_reader($client, $backend)); + + $reqstate->{log}->{code} = 101; + $self->log_request($reqstate); + }, + ); + }; + + return; +} + +# Forward a request verbatim to a service on loopback, unlike proxy_request, +# which re-encodes parsed parameters for another PVE node. The backend is a +# foreign HTTP server, kept behind this server's TLS and authentication. +sub local_http_proxy_request { + my ($self, $reqstate, $method, $target) = @_; + + my $r = $reqstate->{request}; + + my $port = $target->{port}; + my $socket = $target->{socket}; + die "local_http_proxy_request: missing port or socket\n" if !$port && !$socket; + my $path = $target->{path} // '/'; + my $scheme = $target->{tls} ? 'https' : 'http'; + + if ($r->header('upgrade')) { + $self->local_http_proxy_upgrade($reqstate, $method, $target); + return; + } + + # Hop-by-hop headers describe the connection they arrived on, and + # Accept-Encoding goes too, so this server can compress the body itself. + my $skip = { + map { $_ => 1 } qw( + connection keep-alive host content-length transfer-encoding + upgrade te trailer proxy-authorization accept-encoding + ) + }; + + # A unix socket has no authority to name, and nothing behind here routes on + # Host anyway. + my $headers = { Host => $port ? "127.0.0.1:$port" : 'localhost' }; + $r->headers->scan(sub { + my ($key, $value) = @_; + $headers->{$key} = $value if !$skip->{ lc($key) }; + }); + + my $content = $r->content; + $headers->{'Content-Length'} = length($content) if length($content); + + my $tls_ctx; + if ($target->{tls}) { + # Loopback, with a certificate no browser sees and no CA signed: there + # is nothing verification could check. + $tls_ctx = AnyEvent::TLS->new(method => 'any', sslv2 => 0, sslv3 => 0, verify => 0); + } + + # AnyEvent::HTTP needs a URL to parse, so a unix backend gets a nominal + # authority and a tcp_connect that ignores it. + my $url = $port ? "$scheme://127.0.0.1:$port$path" : "$scheme://localhost$path"; + my $tcp_connect; + if ($socket) { + $tcp_connect = sub { + my (undef, undef, $connect_cb, $prepare_cb) = @_; + return AnyEvent::Socket::tcp_connect('unix/', $socket, $connect_cb, $prepare_cb); + }; + } + + my $w; + $w = http_request( + $method => $url, + headers => $headers, + $tcp_connect ? (tcp_connect => $tcp_connect) : (), + timeout => 30, + proxy => undef, # avoid use of $ENV{HTTP_PROXY} + persistent => 0, + keepalive => 0, + body => length($content) ? $content : undef, + $tls_ctx ? (tls_ctx => $tls_ctx) : (), + sub { + my ($body, $hdr) = @_; + + undef $w; + + if (!$reqstate->{hdl}) { + warn "local http proxy detected vanished client connection\n"; + return; + } + + eval { + my $code = delete $hdr->{Status}; + my $msg = delete $hdr->{Reason}; + delete $hdr->{URL}; + delete $hdr->{HTTPVersion}; + + # AnyEvent::HTTP reports its own failures in the 59x range. + if ($code >= 590) { + $self->error($reqstate, HTTP_BAD_GATEWAY, "$msg"); + return; + } + + # Set by this server for the connection it answers on. + delete $hdr->{$_} for qw(connection transfer-encoding content-length); + + my $header = HTTP::Headers->new(%$hdr); + my $resp = HTTP::Response->new($code, $msg, $header, $body); + # Note: disable compression, the backend decides its own encoding + $self->response($reqstate, $resp, undef, 1); + }; + warn $@ if $@; + }, + ); + + return; +} + sub proxy_request { my ($self, $reqstate, $clientip, $host, $node, $method, $uri, $auth, $params) = @_; @@ -1222,6 +1466,25 @@ sub handle_request { # we re-enable timeout in response() $reqstate->{hdl}->timeout(0); + # The handler says where to send it, or nothing for the usual dispatch. + if (my $handler = $self->{local_http_proxy_handler}) { + my $target = eval { $handler->($self, $reqstate, $auth, $method, $path) }; + if (my $err = $@) { + # The handler's refusals are answers: a denial must stay 403. + my $code = HTTP_INTERNAL_SERVER_ERROR; + if (ref($err) && eval { $err->{code} }) { + my $carried = $err->{code}; + $code = $carried if $carried =~ m/^\d+$/ && $carried >= 400 && $carried <= 599; + } + $self->error($reqstate, $code, "$err"); + return; + } + if ($target) { + $self->local_http_proxy_request($reqstate, $method, $target); + return; + } + } + if ($path =~ m/^\Q$base_uri\E/) { $self->handle_api2_request($reqstate, $auth, $method, $path); return; -- 2.55.0