From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id B5CE71FF141 for ; Mon, 16 Mar 2026 13:01:01 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1571317C13; Mon, 16 Mar 2026 13:01:13 +0100 (CET) Date: Mon, 16 Mar 2026 13:00:35 +0100 From: Stoiko Ivanov To: Dominik Csapak Subject: Re: [PATCH pmg-api] fix #7407: cluster: report: increase maximal filesize to 2M Message-ID: <20260316130035.13061a47@rosa.proxmox.com> In-Reply-To: References: <20260313201321.18589-1-s.ivanov@proxmox.com> X-Mailer: Claws Mail 4.3.1 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1773662396160 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.984 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.408 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.819 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.903 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RRD4VCX7E7RCYDI3F2DYDN3YAE2MQQM4 X-Message-ID-Hash: RRD4VCX7E7RCYDI3F2DYDN3YAE2MQQM4 X-MailFrom: s.ivanov@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pmg-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Mail Gateway development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Thanks for the feedback! On Mon, 16 Mar 2026 11:53:58 +0100 Dominik Csapak wrote: > just my thoughts: > > if we're already increasing the limit here, and the file is > already a 1.4M, wouldn't it possibly make sense to increase > the allowed size a bit more? (e.g. 4/8M ?) I think that choice was the reason for sending it to list instead of direclty applying - as I wasn't sure either if 2M or something larger... > > It's not an operation that happens too often (every few minutes) > so the additional memory usage should not matter much. As it's not pre-allocated it matters even less - and if an installation needs it, it's easier to raise the PMG installation's memory than to have to patch the source again... > > Though I guess this doesn't happen often either way, > but setting the limit just a bit over the reported > size, I'm afraid we'll likely run into that again rather > sooner than later. > > IMO in an ideal world we'd never fail the sync due to file size > especially because the INotify code doesn't care about it either. Would consider this once we run into the issue with the increased size of 4M - as then we should probably also rework the diff-detection (currently it's a string-comparison, which might be a bit expensive) in that case (for the cluster-sync). will resent with 4M for now. > > On 3/13/26 9:13 PM, Stoiko Ivanov wrote: > > the postfix configuration files are read via PVE::INotify in our > > stack and have not implicit limit. > > > > PMG::Cluster and PMG::Report do read the same file's contents using > > `file_get_contents` with a limit (explicitly set, or implicit) of 1M. > > > > This causes the cluster-sync to fail in one reported deployment > > hosting ~40k domains, with explicit transport entries and comments > > (resulting in a 1.4M /etc/pmg/transport). > > > > As it's the first occurrence of this causing an issue doubling the > > limit to 2M, instead of increasing it further seems appropriate. > > > > Signed-off-by: Stoiko Ivanov > > --- > > src/PMG/Cluster.pm | 4 ++-- > > src/PMG/Report.pm | 2 +- > > 2 files changed, 3 insertions(+), 3 deletions(-) > > > > diff --git a/src/PMG/Cluster.pm b/src/PMG/Cluster.pm > > index 02d727ee..380cfa6e 100644 > > --- a/src/PMG/Cluster.pm > > +++ b/src/PMG/Cluster.pm > > @@ -266,10 +266,10 @@ my $cond_commit_synced_file = sub { > > return; > > } > > > > - my $new = PVE::Tools::file_get_contents($srcfn, 1024 * 1024); > > + my $new = PVE::Tools::file_get_contents($srcfn, 2048 * 1024); > > > > if (-f $dstfn) { > > - my $old = PVE::Tools::file_get_contents($dstfn, 1024 * 1024); > > + my $old = PVE::Tools::file_get_contents($dstfn, 2048 * 1024); > > return 0 if $new eq $old; > > } > > > > diff --git a/src/PMG/Report.pm b/src/PMG/Report.pm > > index 84d1b887..20e4055d 100644 > > --- a/src/PMG/Report.pm > > +++ b/src/PMG/Report.pm > > @@ -72,7 +72,7 @@ sub dir2text { > > sub { > > my ($file) = @_; > > $report .= "\n# cat $target_dir$file\n"; > > - $report .= PVE::Tools::file_get_contents($target_dir . $file) . "\n"; > > + $report .= PVE::Tools::file_get_contents($target_dir . $file, 2048 * 1024) . "\n"; > > }, > > ); > > } >