[pve-devel] [PATCH http-server v3] http: support Content-Encoding=deflate

[pve-devel] [PATCH http-server v3] http: support Content-Encoding=deflate

[Maximiliano Sandoval]

Add support for compressing the body of responses with
`Content-Encoding: deflate` following [RFC9110]. Note that in this
context `deflate` is actually a "zlib" data format as defined in
[RFC1950].

To preserve the current behavior we prefer `Content-Encoding: gzip`
whenever `gzip` is listed as one of the encodings in the
`Accept-Encoding` header and the data should be compressed.

[RFC9110] https://www.rfc-editor.org/rfc/rfc9110#name-deflate-coding
[RFC1950] https://www.rfc-editor.org/rfc/rfc1950

Suggested-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
---

Differences from v2:
 - Remove redundant post-ifs
 - Add Tested-by: Folke Gleumes trailer

Differences from v1:
 - The commit documents the behavior better
 - Add Suggested-by
 - We set both gzip and deflate in the Accept-Encoding header if available

 src/PVE/APIServer/AnyEvent.pm | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
index b60b825..a8d60c1 100644
--- a/src/PVE/APIServer/AnyEvent.pm
+++ b/src/PVE/APIServer/AnyEvent.pm
@@ -123,6 +123,7 @@ sub cleanup_reqstate {
     delete $reqstate->{request};
     delete $reqstate->{proto};
     delete $reqstate->{accept_gzip};
+    delete $reqstate->{accept_deflate};
     delete $reqstate->{starttime};
 
     if ($reqstate->{tmpfilename}) {
@@ -288,7 +289,7 @@ sub response {
     $reqstate->{hdl}->timeout($self->{timeout});
 
     $nocomp = 1 if !$self->{compression};
-    $nocomp = 1 if !$reqstate->{accept_gzip};
+    $nocomp = 1 if !$reqstate->{accept_gzip} && !$reqstate->{accept_deflate};
 
     my $code = $resp->code;
     my $msg = $resp->message || HTTP::Status::status_message($code);
@@ -333,11 +334,17 @@ sub response {
 	$content_length = length($content);
 
 	if (!$nocomp && ($content_length > 1024)) {
-	    my $comp = Compress::Zlib::memGzip($content);
-	    $resp->header('Content-Encoding', 'gzip');
-	    $content = $comp;
-	    $content_length = length($content);
+	    if ($reqstate->{accept_gzip}) {
+		my $comp = Compress::Zlib::memGzip($content);
+		$resp->header('Content-Encoding', 'gzip');
+		$content = $comp;
+	    } elsif ($reqstate->{accept_deflate}) {
+		my $comp = Compress::Zlib::compress($content);
+		$resp->header('Content-Encoding', 'deflate');
+		$content = $comp;
+	    }
 	}
+	$content_length = length($content);
 	$resp->header("Content-Length" => $content_length);
 	$reqstate->{log}->{content_length} = $content_length;
 
@@ -735,7 +742,15 @@ sub proxy_request {
 	    if $auth->{api_token};
 	$headers->{'CSRFPreventionToken'} = $auth->{token}
 	    if $auth->{token};
-	$headers->{'Accept-Encoding'} = 'gzip' if ($reqstate->{accept_gzip} && $self->{compression});
+	if ($self->{compression}) {
+	    if ($reqstate->{accept_deflate} && $reqstate->{accept_gzip}) {
+		$headers->{'Accept-Encoding'} = 'gzip, deflate';
+	    } elsif ($reqstate->{accept_gzip}) {
+		$headers->{'Accept-Encoding'} = 'gzip';
+	    } elsif ($reqstate->{accept_deflate}) {
+		$headers->{'Accept-Encoding'} = 'deflate';
+	    }
+	}
 
 	if (defined(my $host = $reqstate->{request}->header('Host'))) {
 	    $headers->{Host} = $host;
@@ -1361,6 +1376,7 @@ sub process_header {
     my $conn = $request->header('Connection');
     my $accept_enc = $request->header('Accept-Encoding');
     $reqstate->{accept_gzip} = ($accept_enc && $accept_enc =~ m/gzip/) ? 1 : 0;
+    $reqstate->{accept_deflate} = ($accept_enc && $accept_enc =~ m/deflate/) ? 1 : 0;
 
     if ($conn) {
 	$reqstate->{keep_alive} = 0 if $conn =~ m/close/oi;
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH http-server v3] http: support Content-Encoding=deflate

[Thomas Lamprecht]

Am 18/04/2024 um 11:16 schrieb Maximiliano Sandoval:
> Add support for compressing the body of responses with
> `Content-Encoding: deflate` following [RFC9110]. Note that in this
> context `deflate` is actually a "zlib" data format as defined in
> [RFC1950].
> 
> To preserve the current behavior we prefer `Content-Encoding: gzip`
> whenever `gzip` is listed as one of the encodings in the
> `Accept-Encoding` header and the data should be compressed.
> 
> [RFC9110] https://www.rfc-editor.org/rfc/rfc9110#name-deflate-coding
> [RFC1950] https://www.rfc-editor.org/rfc/rfc1950
> 
> Suggested-by: Lukas Wagner <l.wagner@proxmox.com>
> Tested-by: Folke Gleumes <f.gleumes@proxmox.com>
> Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
> ---
> 
> Differences from v2:
>  - Remove redundant post-ifs
>  - Add Tested-by: Folke Gleumes trailer

thanks, but please add new trailers always to the bottom of the existing
trailer list.

> 
> Differences from v1:
>  - The commit documents the behavior better
>  - Add Suggested-by
>  - We set both gzip and deflate in the Accept-Encoding header if available
> 
>  src/PVE/APIServer/AnyEvent.pm | 28 ++++++++++++++++++++++------
>  1 file changed, 22 insertions(+), 6 deletions(-)
> 
>

applied, thanks!


While I applied this, I still got some questions:
- this uses the default compression level [0], was this chosen by default,
  did you test throughput of the various different levels, e.g. with some
  bigger API call to see if it would be worth it to go for a lower or higher
  level – doing a rough evaluation of this might still be good.

- you use the compression interface, not the deflate one, the former talks
  about RFC 1950 data streams, which often are using the deflate algorithm
  but, as per that RFC, do not necessarily have to.
  Why not use the deflate interface to ensure it's always using the correct
  algorithm?

[0]: https://metacpan.org/pod/Compress::Zlib#COMPRESS/UNCOMPRESS


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel