From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6ED009411F for ; Wed, 12 Apr 2023 16:23:25 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 472A51A357 for ; Wed, 12 Apr 2023 16:22:55 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 12 Apr 2023 16:22:54 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id EB1B3470C0 for ; Wed, 12 Apr 2023 16:22:53 +0200 (CEST) From: Matthias Heiserer To: pve-devel@lists.proxmox.com Date: Wed, 12 Apr 2023 16:22:48 +0200 Message-Id: <20230412142248.184475-1-m.heiserer@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.502 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_NUMSUBJECT 0.5 Subject ends in numbers excluding current years SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH http-server] file upload: don't calculate MD5 X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Apr 2023 14:23:25 -0000 Until now, we calculated the MD5 hash of any uploaded file during the upload, regardless of whether the user chose to provide a hash sum and algorithm. The hash was only logged in the syslog. As the user can provide a hash algorithm and a checksum when uploading a file, which gets automatically checked (after the upload), this is not needed anymore. Instead, the file name is logged. Depending on the speed of the network and the cpu, upload speed or CPU usage might improve: All tests were made by uploading a 3.6GB iso from the PVE host to a local VM. First line is with md5, second without. no networklimit multipart upload complete (size: 3826831360B time: 20.310s rate: 179.69MiB/s md5sum: 8c651682056205967d530697c98d98c3) multipart upload complete (size: 3826831360B time: 16.169s rate: 225.72MiB/s filename: ubuntu-22.04.1-desktop-amd64.iso) 125MB/s network In this test, pveproxy worker used x % CPU during the upload. As you can see, the reduced CPU usage is noticable in slower networks. ~75% CPU: multipart upload complete (size: 3826831360B time: 30.764s rate: 118.63MiB/s md5sum: 8c651682056205967d530697c98d98c3) ~60% CPU: multipart upload complete (size: 3826831360B time: 30.763s rate: 118.64MiB/s filename: ubuntu-22.04.1-desktop-amd64.iso) qemu64 cpu, no network limit multipart upload complete (size: 3826831360B time: 46.113s rate: 79.14MiB/s md5sum: 8c651682056205967d530697c98d98c3) multipart upload complete (size: 3826831360B time: 41.492s rate: 87.96MiB/s filename: ubuntu-22.04.1-desktop-amd64.iso) qemu64, -aes, 1 core, 0.7 cpu multipart upload complete (size: 3826831360B time: 79.875s rate: 45.69MiB/s md5sum: 8c651682056205967d530697c98d98c3) multipart upload complete (size: 3826831360B time: 66.364s rate: 54.99MiB/s filename: ubuntu-22.04.1-desktop-amd64.iso) Signed-off-by: Matthias Heiserer --- src/PVE/APIServer/AnyEvent.pm | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm index ac48899..2fa74d2 100644 --- a/src/PVE/APIServer/AnyEvent.pm +++ b/src/PVE/APIServer/AnyEvent.pm @@ -1245,15 +1245,14 @@ sub file_upload_multipart { if ($write_length > 0) { syswrite($rstate->{outfh}, $data) == $write_length or die "write to temporary file failed - $!\n"; $rstate->{bytes} += $write_length; - $rstate->{ctx}->add($data); } } if ($rstate->{phase} == 100) { # Phase 100 - transfer finished - $rstate->{md5sum} = $rstate->{ctx}->hexdigest; my $elapsed = tv_interval($rstate->{starttime}); - syslog('info', "multipart upload complete (size: %dB time: %.3fs rate: %.2fMiB/s md5sum: %s)", - $rstate->{bytes}, $elapsed, $rstate->{bytes} / ($elapsed * 1024 * 1024), $rstate->{md5sum} + syslog('info', "multipart upload complete (size: %dB time: %.3fs rate: %.2fMiB/s filename: %s)", + $rstate->{bytes}, $elapsed, $rstate->{bytes} / ($elapsed * 1024 * 1024), + $rstate->{params}->{filename} ); $self->handle_api2_request($reqstate, $auth, $method, $path, $rstate); } @@ -1563,7 +1562,6 @@ sub authenticate_and_handle_request { my $state = { size => $len, boundary => $boundary, - ctx => Digest::MD5->new, boundlen => $boundlen, maxheader => 2048 + $boundlen, # should be large enough params => decode_urlencoded($request->url->query()), -- 2.30.2