From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 443341FF163 for ; Thu, 10 Oct 2024 17:25:56 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 332291D443; Thu, 10 Oct 2024 17:26:24 +0200 (CEST) Message-ID: <8c0826e5-8829-469d-9b88-2059aa36fc7a@proxmox.com> Date: Thu, 10 Oct 2024 17:25:50 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Thomas Lamprecht , Proxmox Backup Server development discussion References: <20241009092031.97601-1-c.ebner@proxmox.com> <43470f91-ba2e-4090-a525-c6b64369ad0a@proxmox.com> Content-Language: en-US, de-DE From: Christian Ebner In-Reply-To: <43470f91-ba2e-4090-a525-c6b64369ad0a@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.023 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH v2 proxmox-backup] partial fix #5560: client: periodically show backup progress X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" On 10/10/24 16:45, Thomas Lamprecht wrote: > Am 09/10/2024 um 11:20 schrieb Christian Ebner: >> Spawn a new tokio task which about every minute displays the >> cumulative progress of the backup for pxar, ppxar or img archive >> streams. Catalog and metadata archive streams are excluded from the >> output for better readability, and because the catalog upload lives >> for the whole upload time, leading to possible temporal >> misalignments in the output. The actual payload data is written via >> the other streams anyway. >> >> Add accounting for uploaded chunks, to distinguish from chunks queued >> for upload, but not actually uploaded yet. >> >> Example output in the backup task log: >> ``` >> ... >> INFO: root.pxar: elapsed 60.00 s, new: 191.446 MiB, reused: 0 B, total: 191.446 MiB, uploaded: 13.021 MiB (compressed 5.327 MiB, average: 222.221 KiB/s) >> INFO: root.pxar: elapsed 120.00 s, new: 191.446 MiB, reused: 0 B, total: 191.446 MiB, uploaded: 27.068 MiB (compressed 11.583 MiB, average: 230.977 KiB/s) >> INFO: root.pxar: elapsed 180.00 s, new: 191.446 MiB, reused: 0 B, total: 191.446 MiB, uploaded: 36.138 MiB (compressed 14.987 MiB, average: 205.58 KiB/s) > > Thx for tackling this, but I'm rather nitpicky with the formatting of > progress reports, so quite a bit commentary w.r.t. that: Fine by me, I did include most of the information based on what seemed of interest to me, but it might be a bit to much information at once, agreed. > I'm not a total fan of those averaged bandwidth indicators, as they often > suggest a slow tool (or uplink) if not much new data has to be sent. > If, it might make a bit more sense to print the bandwidth of the total > processed data? Well, this is the average over the cumulative upload, not the progress of each individual time frame. So maybe calling this `total bandwidth` might be more fitting? > Printing the elapsed time just in seconds can be rather unwieldy for longer > running operations, e.g. "elapsed 32280 s, ..." for "8 h 58 m" is not so > easy to parse. A HumanDuration which renders to something like, for example, > "1w 2d 3h 4m 5.67s" could be nicer here (parts that are 0 simply omitted), > but even just a local fn that handles this up to hour range would be a lot > better. Yeah, since the output is generated roughly once a minute, that should help for readability. Will see to improve this in a new version. > And I see some confusion potential with "new" as in "is it new since last > status report output or total new data compared to previous snapshot" All values are given in a cumulative way, non of them is calculated on a time frame difference, seems this needs to be emphasized more in the log output somehow. Above is new in the sense of total bytes of data located in new chunks, to be (or already) uploaded to the server, while the reused indicates the total number of bytes in reused chunks, therefore not re-uploaded. > Is "total" the amount of read data here? As that might be one of the better > indicators, i.e. if I (roughly) know that directory I back up holds > 10 GB of data and the client reports it read 8.7 GB it would be helpful > for me even if it's naturally also not guaranteed to progress linearly. The total is the total number of bytes after chunking, (so the stream length), meaning the sum of bytes of new and reused chunks. > Potentially also just report the compressed amount for "uploaded", as > that's what really got uploaded? The compressed is exactly that, the number of total compressed uploaded bytes. That was the intention of placing it in brackets next to the uploaded value. > As of is, the format seems to benefit devs and technical users the most > way, for the ordinary user it might be a bit much. > > Maybe reduce this to something like: > > processed X data in T (optionally: processing-rate) uploaded Y Okay, that should be a good enough as starting point. The main motivation for this was anyhow to give some feedback to the user that the backup is still making progress. > where X is totally processed data, T is the elapsed time and Y is the > amount of data that actually had to be sent over the network link. > Just as an more actionable idea, there might be better variants. > > FWIW: We could still add a more detail reporting mode enabled through > some CLI option later. Okay, thanks for your input, I will see to improve this in a new version of the patches based on your comments and suggestions. _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel