* [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf
@ 2023-07-28 13:28 Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 manager 1/4] fix #3069: vzdump: add property 'performance: pbs-entries-max=N' Alexander Zeidler
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Alexander Zeidler @ 2023-07-28 13:28 UTC (permalink / raw)
To: pve-devel
Users reported[1] about failing container backups to PBS when having a large amount of files in a directory.
proxmox-backup-client has already 'entries-max' implemented. The default value prevents out of memory (OOM) situations. Overriding this parameter needs at least 'Sys.Modify' permissions.
This patch series makes the new optional performance parameter 'pbs-entries-max' available in vzdump and vzdump.conf.
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=3069
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
* apply Fiona's comments. For details see patch emails.
Changes since v2:
* apply Fiona's comments. For details see patch emails.
* add new commit (pve-docs)
* update coverletter text
Changes since v1:
* apply Fiona's comments. For details see patch emails.
pve-manager:
Alexander Zeidler (1):
fix #3069: vzdump: add property 'performance: pbs-entries-max=N'
configs/vzdump.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
pve-container:
Alexander Zeidler (1):
fix #3069: vzdump: add 'pbs-entries-max' parameter
src/PVE/VZDump/LXC.pm | 6 ++++++
1 file changed, 6 insertions(+)
pve-guest-common:
Alexander Zeidler (1):
fix #3069: vzdump: schema: add 'pbs-entries-max' property
src/PVE/VZDump/Common.pm | 9 +++++++++
1 file changed, 9 insertions(+)
pve-docs:
Alexander Zeidler (1):
vzdump: mention new parameter 'pbs-entries-max'
vzdump.adoc | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
2.39.2
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 manager 1/4] fix #3069: vzdump: add property 'performance: pbs-entries-max=N'
2023-07-28 13:28 [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
@ 2023-07-28 13:28 ` Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 container 2/4]: fix #3069: vzdump: add 'pbs-entries-max' parameter Alexander Zeidler
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Alexander Zeidler @ 2023-07-28 13:28 UTC (permalink / raw)
To: pve-devel
configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
Apply Fiona's comment:
* use only one performance line
Changes since v2:
Apply Fiona's comments:
* remove property from permission check since it's included in performance
* vzdump: make property a sub property of performance
Changes since v1:
* add 'pbs-entries-max' according to the new shortened permission check
configs/vzdump.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/configs/vzdump.conf b/configs/vzdump.conf
index 2ea09ae0..33893e69 100644
--- a/configs/vzdump.conf
+++ b/configs/vzdump.conf
@@ -5,7 +5,7 @@
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
-#performance: max-workers=N
+#performance: [max-workers=N][,pbs-entries-max=N]
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
--
2.39.2
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 container 2/4]: fix #3069: vzdump: add 'pbs-entries-max' parameter
2023-07-28 13:28 [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 manager 1/4] fix #3069: vzdump: add property 'performance: pbs-entries-max=N' Alexander Zeidler
@ 2023-07-28 13:28 ` Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 guest-common 3/4] fix #3069: vzdump: schema: add 'pbs-entries-max' property Alexander Zeidler
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Alexander Zeidler @ 2023-07-28 13:28 UTC (permalink / raw)
To: pve-devel
configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
Apply Fiona's comment:
* fix style (https://lists.proxmox.com/pipermail/pve-devel/2023-January/055365.html)
Changes since v2:
Apply Fiona's comments:
* fix parameter check
* fix both style nits
Changes since v1:
* reword loginfo message and include new set value
* fix indentation
src/PVE/VZDump/LXC.pm | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/src/PVE/VZDump/LXC.pm b/src/PVE/VZDump/LXC.pm
index 5783ffa..c68a06f 100644
--- a/src/PVE/VZDump/LXC.pm
+++ b/src/PVE/VZDump/LXC.pm
@@ -395,6 +395,12 @@ sub archive {
push @$param, '--backup-id', $vmid;
push @$param, '--backup-time', $task->{backup_time};
+ if (my $entries_max = $opts->{"performance"}->{"pbs-entries-max"}) {
+ push $param->@*, '--entries-max', $entries_max;
+ $self->loginfo(
+ "set max number of entries in memory for file-based backups to $entries_max");
+ }
+
my @storage = ($opts->{scfg}, $opts->{storage});
my $logfunc = sub { my $line = shift; $self->loginfo($line) };
--
2.39.2
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 guest-common 3/4] fix #3069: vzdump: schema: add 'pbs-entries-max' property
2023-07-28 13:28 [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 manager 1/4] fix #3069: vzdump: add property 'performance: pbs-entries-max=N' Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 container 2/4]: fix #3069: vzdump: add 'pbs-entries-max' parameter Alexander Zeidler
@ 2023-07-28 13:28 ` Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 docs 4/4] vzdump: mention new parameter 'pbs-entries-max' Alexander Zeidler
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Alexander Zeidler @ 2023-07-28 13:28 UTC (permalink / raw)
To: pve-devel
configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
Apply Fiona's comments:
* simplify description
Changes since v2:
Apply Fiona's comments:
* rephrase description
* fix dot style nit
Changes since v1:
* improve description
* move description to 'performance' section
* remove arrow alignment
src/PVE/VZDump/Common.pm | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 31e9fdb..be605af 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -88,6 +88,15 @@ PVE::JSONSchema::register_format('backup-performance', {
default => 16,
optional => 1,
},
+ 'pbs-entries-max' => {
+ description => "Applies to container backups sent to PBS. Limits the number of entries "
+ ."allowed in memory at a given time to avoid unintended OOM situations. Increase it to "
+ ."enable backups of containers with a large amount of files.",
+ type => 'integer',
+ minimum => 1,
+ default => 1048576,
+ optional => 1,
+ },
});
my $confdesc = {
--
2.39.2
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 docs 4/4] vzdump: mention new parameter 'pbs-entries-max'
2023-07-28 13:28 [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
` (2 preceding siblings ...)
2023-07-28 13:28 ` [pve-devel] [PATCH v4 guest-common 3/4] fix #3069: vzdump: schema: add 'pbs-entries-max' property Alexander Zeidler
@ 2023-07-28 13:28 ` Alexander Zeidler
2023-09-11 11:38 ` [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
2023-09-18 15:05 ` [pve-devel] applied-series: " Thomas Lamprecht
5 siblings, 0 replies; 7+ messages in thread
From: Alexander Zeidler @ 2023-07-28 13:28 UTC (permalink / raw)
To: pve-devel
configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
* none
Changes since v2:
* add this commit
vzdump.adoc | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/vzdump.adoc b/vzdump.adoc
index f3eadcd..f86c91c 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -218,7 +218,8 @@ There are a few settings for tuning backup performance not exposed in the UI.
The most notable is `bwlimit` for limiting IO bandwidth. The amount of threads
used for the compressor can be controlled with the `pigz` (replacing `gzip`),
respectively, `zstd` setting. Furthermore, there are `ionice` and, as part of
-the `performance` setting, `max-workers` (affects VM backups only). See the
+the `performance` setting, `max-workers` (affects VM backups only) and
+`pbs-entries-max` (affects container backups only). See the
xref:vzdump_configuration[configuration options] for details.
[[vzdump_retention]]
--
2.39.2
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf
2023-07-28 13:28 [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
` (3 preceding siblings ...)
2023-07-28 13:28 ` [pve-devel] [PATCH v4 docs 4/4] vzdump: mention new parameter 'pbs-entries-max' Alexander Zeidler
@ 2023-09-11 11:38 ` Alexander Zeidler
2023-09-18 15:05 ` [pve-devel] applied-series: " Thomas Lamprecht
5 siblings, 0 replies; 7+ messages in thread
From: Alexander Zeidler @ 2023-09-11 11:38 UTC (permalink / raw)
To: pve-devel
On Fri, 2023-07-28 at 15:28 +0200, Alexander Zeidler wrote:
> Users reported[1] about failing container backups to PBS when having
> a large amount of files in a directory.
>
> proxmox-backup-client has already 'entries-max' implemented. The
> default value prevents out of memory (OOM) situations. Overriding
> this parameter needs at least 'Sys.Modify' permissions.
>
> This patch series makes the new optional performance parameter 'pbs-
> entries-max' available in vzdump and vzdump.conf.
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=3069
>
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
Ping. Still applies on master.
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] applied-series: [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf
2023-07-28 13:28 [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
` (4 preceding siblings ...)
2023-09-11 11:38 ` [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
@ 2023-09-18 15:05 ` Thomas Lamprecht
5 siblings, 0 replies; 7+ messages in thread
From: Thomas Lamprecht @ 2023-09-18 15:05 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexander Zeidler
Am 28/07/2023 um 15:28 schrieb Alexander Zeidler:
> Users reported[1] about failing container backups to PBS when having a large amount of files in a directory.
>
> proxmox-backup-client has already 'entries-max' implemented. The default value prevents out of memory (OOM) situations. Overriding this parameter needs at least 'Sys.Modify' permissions.
>
> This patch series makes the new optional performance parameter 'pbs-entries-max' available in vzdump and vzdump.conf.
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=3069
>
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> Changes since v3:
> * apply Fiona's comments. For details see patch emails.
>
> Changes since v2:
> * apply Fiona's comments. For details see patch emails.
> * add new commit (pve-docs)
> * update coverletter text
>
> Changes since v1:
> * apply Fiona's comments. For details see patch emails.
>
>
> pve-manager:
> Alexander Zeidler (1):
> fix #3069: vzdump: add property 'performance: pbs-entries-max=N'
>
> configs/vzdump.conf | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>
> pve-container:
> Alexander Zeidler (1):
> fix #3069: vzdump: add 'pbs-entries-max' parameter
>
> src/PVE/VZDump/LXC.pm | 6 ++++++
> 1 file changed, 6 insertions(+)
>
>
> pve-guest-common:
> Alexander Zeidler (1):
> fix #3069: vzdump: schema: add 'pbs-entries-max' property
>
> src/PVE/VZDump/Common.pm | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
>
> pve-docs:
> Alexander Zeidler (1):
> vzdump: mention new parameter 'pbs-entries-max'
>
> vzdump.adoc | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
applied series, thanks!
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-09-18 15:06 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-28 13:28 [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 manager 1/4] fix #3069: vzdump: add property 'performance: pbs-entries-max=N' Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 container 2/4]: fix #3069: vzdump: add 'pbs-entries-max' parameter Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 guest-common 3/4] fix #3069: vzdump: schema: add 'pbs-entries-max' property Alexander Zeidler
2023-07-28 13:28 ` [pve-devel] [PATCH v4 docs 4/4] vzdump: mention new parameter 'pbs-entries-max' Alexander Zeidler
2023-09-11 11:38 ` [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf Alexander Zeidler
2023-09-18 15:05 ` [pve-devel] applied-series: " Thomas Lamprecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox