From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pdm-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id EDDAE1FF15C for <inbox@lore.proxmox.com>; Fri, 18 Apr 2025 09:13:12 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5C8D7372B6; Fri, 18 Apr 2025 09:13:10 +0200 (CEST) Date: Fri, 18 Apr 2025 09:12:36 +0200 From: Wolfgang Bumiller <w.bumiller@proxmox.com> To: Thomas Lamprecht <t.lamprecht@proxmox.com> Message-ID: <b63uc4pklvib5rxdiga3nuopxohiqxpebtakbjl6riqseeognw@6ur22hzflvby> References: <20250417132256.282624-1-l.wagner@proxmox.com> <31497f93-68ac-4bfb-8f42-636edce9cf9f@proxmox.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <31497f93-68ac-4bfb-8f42-636edce9cf9f@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.081 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pdm-devel] [PATCH proxmox-datacenter-manager v3 0/6] remote task cache fetching task / better cache backend X-BeenThere: pdm-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Datacenter Manager development discussion <pdm-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pdm-devel>, <mailto:pdm-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pdm-devel/> List-Post: <mailto:pdm-devel@lists.proxmox.com> List-Help: <mailto:pdm-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel>, <mailto:pdm-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox Datacenter Manager development discussion <pdm-devel@lists.proxmox.com> Cc: Proxmox Datacenter Manager development discussion <pdm-devel@lists.proxmox.com> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: pdm-devel-bounces@lists.proxmox.com Sender: "pdm-devel" <pdm-devel-bounces@lists.proxmox.com> On Thu, Apr 17, 2025 at 05:31:56PM +0200, Thomas Lamprecht wrote: > Am 17.04.25 um 15:22 schrieb Lukas Wagner: > > Benchmarking was done using the 'fake-remote' feature. There were 100 > > remotes, 10 PVE nodes per remote. The task cache contained > > about 1.5 million tasks. > > before after > > list of active tasks (*): ~1.3s ~30=B5s > > list of 500 tasks, offset 0 (**): ~1.3s ~500=B5s > > list of 500 tasks, offset 1 million (***): ~1.3s ~200ms > > Size on disk: ~500MB ~200MB > > = > > (*): Requested by the UI every 3s > > (**): Requested by the UI when visiting Remotes > Tasks > > (***): E.g. when scrolling towars the bottom of 'Remotes > Tasks' > = > Did you regenerate these for the current revision? Just out of interest > whether there is any measurable effect of Wolfgangs proposed changes. > = > And FWIW, I'd like to encode these also in the commit message, albeit > with the lore links that b4 adds it's less of a problem, but IMO would > still not really hurt to do. Depending on your answer on the actuality > of above data I could amend that or newly provided measurements into > the commit message on applying though, so definitively no need for a v+1 > just for that. > = > > = > > In the old implementation, the archive file was *always* fully deserial= ized > > and loaded into RAM, this is the reason why the time needed is pretty > > idential for all scenarios. > > The new implementation reads the archive files only line by line, > > and only 500 tasks were loaded into RAM at the same time. The higher th= e offset, > > the more archive lines/files we have to scan, which increases the > > time needed to access the data. The tasks are sorted descending > > by starttime, as a result the requests get slower the further you > > go back in history. > > = > > The 'before' times do NOT include the time needed for actually fetching > > the task data. > > = > > This series was preseded by [1], however almost all of the code has cha= nges, which > > is the reason why I send this as a new series. > > = > > [1] https://lore.proxmox.com/pdm-devel/20250128122520.167796-1-l.wagner= @proxmox.com/ > > = > > Changes since v2: > > - Change locking approach as suggested by Wolfgang > > - Incorporated feedback from Wolfang > > - see patch notes for details > > - Added some .context/.with_context for better error messages > > = > = > Looks alright to me now, if Wolfgang sees nothing of w.r.t. the changes > since the last version I'd apply this series tomorrow. Mostly LGTM, just minor issues. (A bunch of `pub`s to drop, one of which is important in patch 3.) _______________________________________________ pdm-devel mailing list pdm-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel