From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 0BD261FF161 for ; Tue, 24 Sep 2024 16:33:25 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0885AF7B2; Tue, 24 Sep 2024 16:33:37 +0200 (CEST) From: Filip Schauer To: pve-devel@lists.proxmox.com Date: Tue, 24 Sep 2024 16:33:21 +0200 Message-Id: <20240924143321.93696-1-f.schauer@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.049 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [tools.pm, proxmox.com] Subject: [pve-devel] [PATCH common] tools: file_set_contents: use syswrite instead of print X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" The use of `print` can be inefficient for writing larger files due to its default buffering in 8 KiB blocks. This is especially problematic on `pmxcfs` where files are written in 4 KiB blocks due to the defaults of `libfuse2`. This leads to significant write amplification on files larger than 4 KiB. Patch (fix #5728: pmxcfs: allow bigger writes than 4k for fuse) [1] addresses this by enabling `big_writes`, allowing up to 128 KiB blocks. But due to the use of `print` in `file_set_contents`, writes are still only buffered in 8 KiB blocks. To further address this, this commit switches to using `syswrite` instead of `print` to mitigate the block size limit imposed by `print`. Combined with patch [1], file writes to `/etc/pve/` are now buffered in 128 KiB blocks. The table below illustrates the drastic reduction in write amplification when writing files of different sizes to `/etc/pve/` using `file_set_contents`: print big_writes+print big_writes+syswrite data size written amplif. written amplif. written amplif. 1 KiB 48 KiB 48.0 45 KiB 45.0 41 KiB 41.0 2 KiB 48 KiB 24.0 45 KiB 22.5 62 KiB 31.0 4 KiB 82 KiB 20.5 80 KiB 20.0 73 KiB 18.3 8 KiB 121 KiB 15.1 90 KiB 11.3 89 KiB 11.1 16 KiB 217 KiB 13.6 146 KiB 9.1 113 KiB 7.1 32 KiB 506 KiB 15.8 314 KiB 9.8 158 KiB 4.9 64 KiB 1472 KiB 23.0 826 KiB 12.9 259 KiB 4.0 128 KiB 5585 KiB 43.6 3765 KiB 29.4 452 KiB 3.5 256 KiB 20424 KiB 79.8 10743 KiB 42.0 2351 KiB 9.2 512 KiB 86715 KiB 169.4 43650 KiB 85.3 3204 KiB 6.3 1024 KiB 369568 KiB 360.9 187496 KiB 183.1 15845 KiB 15.5 [1] https://lists.proxmox.com/pipermail/pve-devel/2024-September/065396.html Signed-off-by: Filip Schauer --- src/PVE/Tools.pm | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm index bd305bd..0edf166 100644 --- a/src/PVE/Tools.pm +++ b/src/PVE/Tools.pm @@ -285,10 +285,17 @@ sub file_set_contents { } die "unable to open file '$tmpname' - $!\n" if !$fh; - binmode($fh, ":encoding(UTF-8)") if $force_utf8; + $data = encode("utf8", $data) if $force_utf8; - die "unable to write '$tmpname' - $!\n" unless print $fh $data; - die "closing file '$tmpname' failed - $!\n" unless close $fh; + my $offset = 0; + my $len = length($data); + + while ($offset < $len) { + $offset += syswrite($fh, $data, $len - $offset, $offset) + or die "unable to write '$tmpname' - $!\n"; + } + + close $fh or die "closing file '$tmpname' failed - $!\n"; }; my $err = $@; -- 2.39.2 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel