From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 947C67D7DE for ; Thu, 21 Jul 2022 14:40:58 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 8BE371802F for ; Thu, 21 Jul 2022 14:40:28 +0200 (CEST) Received: from picard.linux.it (picard.linux.it [IPv6:2001:1418:10:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 21 Jul 2022 14:40:27 +0200 (CEST) Received: by picard.linux.it (Postfix, from userid 10) id DEE0F3C98CE; Thu, 21 Jul 2022 14:40:20 +0200 (CEST) Received: from news by eraldo.lilliput.linux.it with local (Exim 4.89) (envelope-from ) id 1oEVPN-0007r4-Te for pve-user@lists.proxmox.com; Thu, 21 Jul 2022 14:36:01 +0200 From: Marco Gaiarin Date: Thu, 21 Jul 2022 14:10:09 +0200 Organization: Il gaio usa sempre TIN per le liste, fallo anche tu!!! Message-ID: X-Trace: eraldo.lilliput.linux.it 1658405434 29682 192.168.24.2 (21 Jul 2022 12:10:34 GMT) X-Mailer: tin/2.4.4-20191224 ("Millburn") (Linux/5.4.0-122-generic (x86_64)) X-Gateway-System: SmartGate 1.4.5 To: pve-user@lists.proxmox.com X-SPAM-LEVEL: Spam detection results: 0 AWL 0.188 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% JMQ_SPF_NEUTRAL 0.5 SPF set to ?all KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [PVE-User] Again on ZFS strange performance... X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2022 12:40:58 -0000 Situation: DELL PowerEdge T440, 64GB RAM, Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz, 16 cores. On the server we have created a 'data' pool, backing it with some SSD cache: root@pppve2:~# zpool status rpool-data pool: rpool-data state: ONLINE config: NAME STATE READ WRITE CKSUM rpool-data ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ata-HGST_HUS726T4TALA6L0_V1KNZMLL ONLINE 0 0 0 ata-HGST_HUS726T4TALA6L0_V1KNY60L ONLINE 0 0 0 ata-HGST_HUS726T4TALA6L0_V1KNY6AL ONLINE 0 0 0 logs ata-MZ7KH480HAHQ0D3_S5CNNA0RA29699-part1 ONLINE 0 0 0 cache ata-MZ7KH480HAHQ0D3_S5CNNA0RA29699-part2 ONLINE 0 0 0 errors: No known data errors On the server we have created a NFS server that use the 'rpool-data' pool space, and we are migrating disks (disks, not VM) from an older PVE infrastructure in this NFS server (and thus, on this pool). Moving disks into the NFS server are done at 'wirespeed' (gigabit), eg 50/60 MByte/s, with little or no load on the destination server: iodelay under 10%, load around 4-6. After moving disks, i've moved manually the .conf machine configuration in the new architecture, the started it. After that, i've moved the disks from the NFS server to the zpools; i've moved the (little) systemd disk from NFS to 'local-zf' (full SSD ZFS pool) in some second. I've tried to move the data disk (2TB) from NFS to 'rpool-data' and: 1) if i setup NO limit on disk move, system get thrashed roughly istantaneous (load 40, iodelay 60-80%). 2) if i put as a limit 25MB/s, i can copy 100G roughly, after that probably the SSD cache get full and i get load 20 and iodelay 50-60%: system is thrashed but still usable. 3) now i've put 10MB/s as a limit, and seems is working, slowly but with no consequences. But is slow... But still i'm a bit confused... why i can write to a pool via NFS at 60MB/s, but i have to move a disk at 10MB/s?! -- ``... La memoria conta veramente solo se tiene insieme l'impronta del presente e il progetto del futuro, se permette di fare senza dimenticare quel che si voleva fare, di diventare senza smettere di essere, di essere senza smettere di diventare...'' (Italo Calvino)