[pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf

[pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf

[Alexander Zeidler]

Users reported[1] about failing container backups to PBS when having a large amount of files in a directory.

proxmox-backup-client has already 'entries-max' implemented. The default value prevents out of memory (OOM) situations. Overriding this parameter needs at least 'Sys.Modify' permissions.

This patch series makes the new optional performance parameter 'pbs-entries-max' available in vzdump and vzdump.conf.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=3069

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
  * apply Fiona's comments. For details see patch emails.

Changes since v2:
  * apply Fiona's comments. For details see patch emails.
  * add new commit (pve-docs)
  * update coverletter text

Changes since v1:
  * apply Fiona's comments. For details see patch emails.


pve-manager:
Alexander Zeidler (1):
  fix #3069: vzdump: add property 'performance: pbs-entries-max=N'

 configs/vzdump.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


pve-container:
Alexander Zeidler (1):
  fix #3069: vzdump: add 'pbs-entries-max' parameter

 src/PVE/VZDump/LXC.pm | 6 ++++++
 1 file changed, 6 insertions(+)


pve-guest-common:
Alexander Zeidler (1):
  fix #3069: vzdump: schema: add 'pbs-entries-max' property

 src/PVE/VZDump/Common.pm | 9 +++++++++
 1 file changed, 9 insertions(+)


pve-docs:
Alexander Zeidler (1):
  vzdump: mention new parameter 'pbs-entries-max'

 vzdump.adoc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
2.39.2

[pve-devel] [PATCH v4 manager 1/4] fix #3069: vzdump: add property 'performance: pbs-entries-max=N'

[Alexander Zeidler]

configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
  Apply Fiona's comment:
  * use only one performance line

Changes since v2:
  Apply Fiona's comments:
  * remove property from permission check since it's included in performance
  * vzdump: make property a sub property of performance

Changes since v1:
  * add 'pbs-entries-max' according to the new shortened permission check


 configs/vzdump.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configs/vzdump.conf b/configs/vzdump.conf
index 2ea09ae0..33893e69 100644
--- a/configs/vzdump.conf
+++ b/configs/vzdump.conf
@@ -5,7 +5,7 @@
 #storage: STORAGE_ID
 #mode: snapshot|suspend|stop
 #bwlimit: KBPS
-#performance: max-workers=N
+#performance: [max-workers=N][,pbs-entries-max=N]
 #ionice: PRI
 #lockwait: MINUTES
 #stopwait: MINUTES
-- 
2.39.2

[pve-devel] [PATCH v4 container 2/4]: fix #3069: vzdump: add 'pbs-entries-max' parameter

[Alexander Zeidler]

configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
  Apply Fiona's comment:
  * fix style (https://lists.proxmox.com/pipermail/pve-devel/2023-January/055365.html)

Changes since v2:
  Apply Fiona's comments:
  * fix parameter check
  * fix both style nits

Changes since v1:
  * reword loginfo message and include new set value
  * fix indentation


 src/PVE/VZDump/LXC.pm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/PVE/VZDump/LXC.pm b/src/PVE/VZDump/LXC.pm
index 5783ffa..c68a06f 100644
--- a/src/PVE/VZDump/LXC.pm
+++ b/src/PVE/VZDump/LXC.pm
@@ -395,6 +395,12 @@ sub archive {
 	push @$param, '--backup-id', $vmid;
 	push @$param, '--backup-time', $task->{backup_time};
 
+	if (my $entries_max = $opts->{"performance"}->{"pbs-entries-max"}) {
+	    push $param->@*, '--entries-max', $entries_max;
+	    $self->loginfo(
+		"set max number of entries in memory for file-based backups to $entries_max");
+	}
+
 	my @storage = ($opts->{scfg}, $opts->{storage});
 
 	my $logfunc = sub { my $line = shift; $self->loginfo($line) };
-- 
2.39.2

[pve-devel] [PATCH v4 guest-common 3/4] fix #3069: vzdump: schema: add 'pbs-entries-max' property

[Alexander Zeidler]

configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
  Apply Fiona's comments:
  * simplify description

Changes since v2:
  Apply Fiona's comments:
  * rephrase description
  * fix dot style nit

Changes since v1:
  * improve description
  * move description to 'performance' section
  * remove arrow alignment


 src/PVE/VZDump/Common.pm | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 31e9fdb..be605af 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -88,6 +88,15 @@ PVE::JSONSchema::register_format('backup-performance', {
 	default => 16,
 	optional => 1,
     },
+    'pbs-entries-max' => {
+	description => "Applies to container backups sent to PBS. Limits the number of entries "
+	    ."allowed in memory at a given time to avoid unintended OOM situations. Increase it to "
+	    ."enable backups of containers with a large amount of files.",
+	type => 'integer',
+	minimum => 1,
+	default => 1048576,
+	optional => 1,
+    },
 });
 
 my $confdesc = {
-- 
2.39.2

[pve-devel] [PATCH v4 docs 4/4] vzdump: mention new parameter 'pbs-entries-max'

[Alexander Zeidler]

configuring pbs-entries-max can avoid failing backups due to a high
amount of files in folders where a folder exclusion is not possible

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v3:
  * none

Changes since v2:
  * add this commit


 vzdump.adoc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/vzdump.adoc b/vzdump.adoc
index f3eadcd..f86c91c 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -218,7 +218,8 @@ There are a few settings for tuning backup performance not exposed in the UI.
 The most notable is `bwlimit` for limiting IO bandwidth. The amount of threads
 used for the compressor can be controlled with the `pigz` (replacing `gzip`),
 respectively, `zstd` setting. Furthermore, there are `ionice` and, as part of
-the `performance` setting, `max-workers` (affects VM backups only). See the
+the `performance` setting, `max-workers` (affects VM backups only) and
+`pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
 [[vzdump_retention]]
-- 
2.39.2

Re: [pve-devel] [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf

[Alexander Zeidler]

On Fri, 2023-07-28 at 15:28 +0200, Alexander Zeidler wrote:
> Users reported[1] about failing container backups to PBS when having
> a large amount of files in a directory.
> 
> proxmox-backup-client has already 'entries-max' implemented. The
> default value prevents out of memory (OOM) situations. Overriding
> this parameter needs at least 'Sys.Modify' permissions.
> 
> This patch series makes the new optional performance parameter 'pbs-
> entries-max' available in vzdump and vzdump.conf.
> 
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=3069
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>

Ping. Still applies on master.

[pve-devel] applied-series: [PATCH v4 manager/container/guest-common/docs 0/4] fix #3069: add pbs-entries-max to vzdump.conf

[Thomas Lamprecht]

Am 28/07/2023 um 15:28 schrieb Alexander Zeidler:
> Users reported[1] about failing container backups to PBS when having a large amount of files in a directory.
> 
> proxmox-backup-client has already 'entries-max' implemented. The default value prevents out of memory (OOM) situations. Overriding this parameter needs at least 'Sys.Modify' permissions.
> 
> This patch series makes the new optional performance parameter 'pbs-entries-max' available in vzdump and vzdump.conf.
> 
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=3069
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> Changes since v3:
>   * apply Fiona's comments. For details see patch emails.
> 
> Changes since v2:
>   * apply Fiona's comments. For details see patch emails.
>   * add new commit (pve-docs)
>   * update coverletter text
> 
> Changes since v1:
>   * apply Fiona's comments. For details see patch emails.
> 
> 
> pve-manager:
> Alexander Zeidler (1):
>   fix #3069: vzdump: add property 'performance: pbs-entries-max=N'
> 
>  configs/vzdump.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> pve-container:
> Alexander Zeidler (1):
>   fix #3069: vzdump: add 'pbs-entries-max' parameter
> 
>  src/PVE/VZDump/LXC.pm | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> 
> pve-guest-common:
> Alexander Zeidler (1):
>   fix #3069: vzdump: schema: add 'pbs-entries-max' property
> 
>  src/PVE/VZDump/Common.pm | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> 
> pve-docs:
> Alexander Zeidler (1):
>   vzdump: mention new parameter 'pbs-entries-max'
> 
>  vzdump.adoc | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 


applied series, thanks!