From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <d.csapak@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 78AB48C457 for <pbs-devel@lists.proxmox.com>; Mon, 31 Oct 2022 12:40:03 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id CD35D16E2 for <pbs-devel@lists.proxmox.com>; Mon, 31 Oct 2022 12:40:02 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pbs-devel@lists.proxmox.com>; Mon, 31 Oct 2022 12:40:01 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0791044BD9 for <pbs-devel@lists.proxmox.com>; Mon, 31 Oct 2022 12:39:55 +0100 (CET) From: Dominik Csapak <d.csapak@proxmox.com> To: pbs-devel@lists.proxmox.com Date: Mon, 31 Oct 2022 12:39:50 +0100 Message-Id: <20221031113953.3111599-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.067 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [RFC PATCH backup-restore-image/proxmox-backup] improve file-restore for zpools X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Mon, 31 Oct 2022 11:40:03 -0000 when using file-restore for guests with a zpool, the minimal memory amount given (up to 384 MiB depending on the disks) is often not enough, resulting in an oom kill of our restore daemon inside the vm the easiest way to trigger that is having many subvolumes in the pool (e.g. i could trigger it easily with ~100 subvolumes) to improve this, we dynamically add more (1GiB) when we see that the client tries to access a zpool with this, i successfully tested up to 1000 subvolumes (altough it took a few minutes to mount, not exactly sure why) for this to work, we have to adapt our custom kernel config a bit, so that we include the memory hotplug configs RFC because: * not sure if the value of 1GiB is sensible, i tried to give some amount that is not too much, but should fix a wide range of cases maybe we want to make that configurable? if we do, maybe we should discard this approach and simply make the general amount of memory configurable? * not sure about the hotplug mechanism, we could use virtio-mem instead, but in the current case i don't see any advantages (we have to predefine the maximum value anyway, and since we only have a few points where we update the memory, we don't need to do it in small increments, it also would require more kernel config flags) * i mostly ignore the QMP answers from qemu because they are mostly empty and hold no real information. we could of course build a complete qmp client in rust (i guess we need that sooner or later anyway) but depending on how exactly we want to parse qemu messages this will increase the amount of work significantly Note that when we want to bump the file-restore package, we still need to apply my previous patches for pve-common/storage [0] patch 1/2 of proxmox-backup is a simple cleanup that could be applied individually 0: https://lists.proxmox.com/pipermail/pve-devel/2022-May/053085.html proxmox-backup-restore-image: Dominik Csapak (1): add kernel config options for memory hotplug src/config-base | 6 ++++++ 1 file changed, 6 insertions(+) proxmox-backup: Dominik Csapak (2): file-restore: fix deprecated qemu parameters file-restore: dynamically increase memory of vm for zpools proxmox-file-restore/src/block_driver_qemu.rs | 45 +++++++++-- proxmox-file-restore/src/qemu_helper.rs | 74 ++++++++++++++++++- 2 files changed, 108 insertions(+), 11 deletions(-) -- 2.30.2