[pve-devel] [PATCH v3 http-server 1/3] multipart upload: fix upload of files starting with newlines

[pve-devel] [PATCH v3 http-server 1/3] multipart upload: fix upload of files starting with newlines

[Matthias Heiserer]

Currently, if a file starts with a newline, it gets removed
and the uploda succeeds (provided no hash is given).

Signed-off-by: Matthias Heiserer <m.heiserer@proxmox.com>
---
 src/PVE/APIServer/AnyEvent.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
index f397a8c..545c122 100644
--- a/src/PVE/APIServer/AnyEvent.pm
+++ b/src/PVE/APIServer/AnyEvent.pm
@@ -1217,7 +1217,7 @@ sub file_upload_multipart {
 	    if ($hdl->{rbuf} =~
 		s/^${delim_re}
 		Content-Disposition:\ (.*?);\ name="(.*?)";\ filename="([^"]+)"${newline_re}
-		Content-Type:\ \S*\s+
+		Content-Type:\ \S*${newline_re}{2}
 		//sxx
 	    ) {
 		assert_form_disposition($1);
-- 
2.30.2

[pve-devel] [PATCH v3 http-server 2/3] fix multipart upload: ignore additional headers

[Matthias Heiserer]

Reported in the forum: https://forum.proxmox.com/threads/image-upload-fails-after-upgrading-from-7-1-to-7-3.119051/#post-516517

When additional headers existed in the request body, the upload failed.
With this patch, all additional headers get ignored.

Example: The following upload would fail because no
headers were expected after Content-Disposition.

```
--EPIHyQJFC5ftgoXHMe8-Jc6E7FqA4oMb0QBfOTz
Content-Disposition: form-data; name="content"
Content-Type: text/plain; charset=ISO-8859-1

iso
```
would fail. These headers now also get ignored, as we don't use them.

Also, upload now works when the Content-Disposition header isn't the first, i.e.
```
--XVH95dt1-A3J8mWiLCmHCW4roSC7-gBntjATBy--
Content-Type: text/plain; charset=ISO-8859-1
Content-Disposition: form-data; name="content"
```

Fixed upload was tested using
* Curl
* GUI
* Apache HttpClient 5

Signed-off-by: Matthias Heiserer <m.heiserer@proxmox.com>
---

Changes from v2:
None

Changes from v1:
* wait for delimiter before handling data, as reported in forum
* squash ignore order of headers patch

 src/PVE/APIServer/AnyEvent.pm | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
index 545c122..2f8718f 100644
--- a/src/PVE/APIServer/AnyEvent.pm
+++ b/src/PVE/APIServer/AnyEvent.pm
@@ -1201,11 +1201,19 @@ sub file_upload_multipart {
 	    $rstate->{phase} = 1;
 	}
 
+	my $remove_until_data = sub {
+	    my ($hdl) = @_;
+	    # remove any remaining multipart "headers" like Content-Type
+	    $hdl->{rbuf} =~ s/^.*?${newline_re}{2}//s;
+	};
+
 	my $extract_form_disposition = sub {
 	    my ($name) = @_;
-	    if ($hdl->{rbuf} =~ s/^${delim_re}Content-Disposition: (.*?); name="$name"(.*?)($delim_re)/$3/s) {
+	    if ($hdl->{rbuf} =~ s/^${delim_re}.*?Content-Disposition: (.*?); name="$name"(.*?${delim_re})/$2/s) {
 		assert_form_disposition($1);
-		$rstate->{params}->{$name} = trim($2);
+		$remove_until_data->($hdl);
+		$hdl->{rbuf} =~ s/^(.*?)(${delim_re})/$2/s;
+		$rstate->{params}->{$name} = trim($1);
 	    }
 	};
 
-- 
2.30.2

[pve-devel] [PATCH v3 http-server 3/3] multipart upload: don't require trailing newline

[Matthias Heiserer]

Allow upload without trailing newline.
This is not compliant with RFC 1521.

RFC 1521 mandates that the close-delimiter ends
in a newline:
'close-delimiter := "--" boundary "--" CRLF'
However, some software (e.g. postman) sends
their request without a trailing newline, which resulted
in failing uploads.

Signed-off-by: Matthias Heiserer <m.heiserer@proxmox.com>
---

Changes from v2:
Reword commit message as I misunderstood the RFC

 src/PVE/APIServer/AnyEvent.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
index 2f8718f..3cd77fa 100644
--- a/src/PVE/APIServer/AnyEvent.pm
+++ b/src/PVE/APIServer/AnyEvent.pm
@@ -1193,7 +1193,7 @@ sub file_upload_multipart {
 
 	my $newline_re = qr/\015?\012/;
 	my $delim_re = qr/--\Q$boundary\E${newline_re}/;
-	my $close_delim_re = qr/--\Q$boundary\E--${newline_re}/;
+	my $close_delim_re = qr/--\Q$boundary\E--/;
 
 	# Phase 0 - preserve boundary, but remove everything before
 	if ($rstate->{phase} == 0 && $hdl->{rbuf} =~ s/^.*?($delim_re)/$1/s) {
-- 
2.30.2

Re: [pve-devel] [PATCH v3 http-server 1/3] multipart upload: fix upload of files starting with newlines

[Daniel Tschlatscher]

Testing this series in the Browser, with curl and postman, I couldn't
find any issues anymore, more details below. Code looks good to me as well.


Tested-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
Reviewed-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>


Browser/GUI:

✅ Uploading files with 0B, 1B, 1kB, 17kB, 1MB, 1GB, 10GB
✅ Uploading file with a SHA256 checksum

In curl and Postman:

✅ Changing the extension in the first boundary (error)
✅ Adding additional headers leading or trailing (ignored)
✅ Specifying no headers in first boundary (error)
✅ Inconsistent boundary parameter in the Content-Type header (error)
✅ Inconsistent boundary in the body (error)
✅ Whitespaces at the beginning of the file are not discarded
✅ Arbitrary input after the last boundary
✅ Nothing after last boundary
✅ Mixed \n and \r\n in body


On 12/12/22 16:07, Matthias Heiserer wrote:
> Currently, if a file starts with a newline, it gets removed
> and the uploda succeeds (provided no hash is given).
> 
> Signed-off-by: Matthias Heiserer <m.heiserer@proxmox.com>
> ---
>  src/PVE/APIServer/AnyEvent.pm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
> index f397a8c..545c122 100644
> --- a/src/PVE/APIServer/AnyEvent.pm
> +++ b/src/PVE/APIServer/AnyEvent.pm
> @@ -1217,7 +1217,7 @@ sub file_upload_multipart {
>  	    if ($hdl->{rbuf} =~
>  		s/^${delim_re}
>  		Content-Disposition:\ (.*?);\ name="(.*?)";\ filename="([^"]+)"${newline_re}
> -		Content-Type:\ \S*\s+
> +		Content-Type:\ \S*${newline_re}{2}
>  		//sxx
>  	    ) {
>  		assert_form_disposition($1);

[pve-devel] applied-series: [PATCH v3 http-server 1/3] multipart upload: fix upload of files starting with newlines

[Thomas Lamprecht]

Am 12/12/2022 um 16:07 schrieb Matthias Heiserer:
> Currently, if a file starts with a newline, it gets removed
> and the uploda succeeds (provided no hash is given).
> 
> Signed-off-by: Matthias Heiserer <m.heiserer@proxmox.com>
> ---
>  src/PVE/APIServer/AnyEvent.pm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
>

applied series with Daniel's R-b and T-b, thanks to both of you!