From: Dominik Csapak <d.csapak@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [RFC PATCH backup-restore-image/proxmox-backup] improve file-restore for zpools
Date: Mon, 31 Oct 2022 12:39:50 +0100 [thread overview]
Message-ID: <20221031113953.3111599-1-d.csapak@proxmox.com> (raw)
when using file-restore for guests with a zpool, the minimal memory
amount given (up to 384 MiB depending on the disks) is often not enough,
resulting in an oom kill of our restore daemon inside the vm
the easiest way to trigger that is having many subvolumes in the pool
(e.g. i could trigger it easily with ~100 subvolumes)
to improve this, we dynamically add more (1GiB) when we see that the
client tries to access a zpool
with this, i successfully tested up to 1000 subvolumes (altough it took
a few minutes to mount, not exactly sure why)
for this to work, we have to adapt our custom kernel config a bit, so
that we include the memory hotplug configs
RFC because:
* not sure if the value of 1GiB is sensible, i tried to give some amount
that is not too much, but should fix a wide range of cases
maybe we want to make that configurable? if we do, maybe we should
discard this approach and simply make the general amount of memory
configurable?
* not sure about the hotplug mechanism, we could use virtio-mem instead,
but in the current case i don't see any advantages
(we have to predefine the maximum value anyway, and since we only
have a few points where we update the memory, we don't need to do it
in small increments, it also would require more kernel config flags)
* i mostly ignore the QMP answers from qemu because they are mostly
empty and hold no real information. we could of course build a
complete qmp client in rust (i guess we need that sooner or later
anyway) but depending on how exactly we want to parse qemu messages
this will increase the amount of work significantly
Note that when we want to bump the file-restore package, we still need
to apply my previous patches for pve-common/storage [0]
patch 1/2 of proxmox-backup is a simple cleanup that could be applied
individually
0: https://lists.proxmox.com/pipermail/pve-devel/2022-May/053085.html
proxmox-backup-restore-image:
Dominik Csapak (1):
add kernel config options for memory hotplug
src/config-base | 6 ++++++
1 file changed, 6 insertions(+)
proxmox-backup:
Dominik Csapak (2):
file-restore: fix deprecated qemu parameters
file-restore: dynamically increase memory of vm for zpools
proxmox-file-restore/src/block_driver_qemu.rs | 45 +++++++++--
proxmox-file-restore/src/qemu_helper.rs | 74 ++++++++++++++++++-
2 files changed, 108 insertions(+), 11 deletions(-)
--
2.30.2
next reply other threads:[~2022-10-31 11:40 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-31 11:39 Dominik Csapak [this message]
2022-10-31 11:39 ` [pbs-devel] [RFC PATCH backup-restore-image 1/1] add kernel config options for memory hotplug Dominik Csapak
2022-11-07 12:48 ` [pbs-devel] applied: " Wolfgang Bumiller
2022-10-31 11:39 ` [pbs-devel] [PATCH proxmox-backup 1/2] file-restore: fix deprecated qemu parameters Dominik Csapak
2022-11-04 12:29 ` [pbs-devel] applied: " Wolfgang Bumiller
2022-10-31 11:39 ` [pbs-devel] [RFC PATCH proxmox-backup 2/2] file-restore: dynamically increase memory of vm for zpools Dominik Csapak
2022-11-07 12:35 ` [pbs-devel] applied: " Wolfgang Bumiller
2022-11-07 17:15 ` Thomas Lamprecht
2022-11-08 7:59 ` Dominik Csapak
2022-11-08 9:19 ` Thomas Lamprecht
2022-11-08 10:07 ` Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221031113953.3111599-1-d.csapak@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox