From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id ACD8D1FF168 for ; Mon, 14 Oct 2024 12:09:11 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B7EC731EC8; Mon, 14 Oct 2024 12:09:41 +0200 (CEST) From: Dominik Csapak To: pve-devel@lists.proxmox.com Date: Mon, 14 Oct 2024 12:09:38 +0200 Message-Id: <20241014100938.1288020-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.016 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH cluster] fix #5728: pmxcfs: allow bigger writes than 4k for fuse X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" by default libfuse2 limits writes to 4k size, which means that on writes bigger than that, we do a whole write cycle for each 4k block that comes in. To avoid that, add the option 'big_writes' to allow writes bigger than 4k at once (namely up to 128 KiB). This means that if we update a file with more than 4KiB data, the following pattern occurs: * cfs_fuse_write is called with at offset 0 with 4096 size * sqlite writes the partial file to disk since it's a transaction * cfs_fuse_write is called with an offset 4096 and with 4096 size * sqlite updates the data and writes again * repeat until all data reached cfs_fuse_write So when cfs_fuse_write accepts bigger chunks, we have less cfs_fuse_write -> sqlite write cycles, leading to a reduced disk activity. Note that sqlite itself uses 4096 byte blocks to write to the file system layer below. Most files on pmxcfs are written with `file_set_contents`, which writes the file into a tmp file and renames it, so we always have some write overhead. Previous to pve-common commit ef0bcc9 (tools: file_set_contents: use syswrite instead of print) it used `print` to write, which uses an internal 8k buffer, and after the commit it uses `syswrite`, which writes the file unbuffered in one go. (Fuse still splits writes at it's defined maximum) The commit message of that patch includes benchmarks for various sizes of writes on pmxcfs with this patch included. Results show that we can reduce the amount of bytes written to disk for files larger than 4 KiB by a significant amount (with both patches we can reduce the amplification at 8KiB from ~15x to ~11x, and for 1024KiB from ~360x to ~15x) When we change to libfuse3, we have to remove this option again, since it got removed and is the default there. Signed-off-by: Dominik Csapak --- changes from rfc: * improve commit message to contain more detail and reference Filips commit that improves `file_set_contents` * add a comment above the option to remove it with change to libfuse3 src/pmxcfs/pmxcfs.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/pmxcfs/pmxcfs.c b/src/pmxcfs/pmxcfs.c index 1cf8ab8..f0d5102 100644 --- a/src/pmxcfs/pmxcfs.c +++ b/src/pmxcfs/pmxcfs.c @@ -945,7 +945,8 @@ int main(int argc, char *argv[]) mkdir(CFSDIR, 0755); - char *fa[] = { "-f", "-odefault_permissions", "-oallow_other", NULL}; + // TODO: remove big_writes with change to libfuse3 + char *fa[] = { "-f", "-odefault_permissions", "-oallow_other", "-obig_writes", NULL}; struct fuse_args fuse_args = FUSE_ARGS_INIT(sizeof (fa)/sizeof(gpointer) - 1, fa); -- 2.39.5 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel