From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 86CBF1FF15C for ; Fri, 28 Nov 2025 11:40:49 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0237117352; Fri, 28 Nov 2025 11:41:11 +0100 (CET) Message-ID: <8e027e83-de92-48d1-a6e4-b1e3f2b80be8@proxmox.com> Date: Fri, 28 Nov 2025 11:40:37 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Proxmox Backup Server development discussion , Robert Obkircher References: <20251121154106.174571-1-r.obkircher@proxmox.com> Content-Language: en-US, de-DE From: Christian Ebner In-Reply-To: <20251121154106.174571-1-r.obkircher@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1764326398953 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.048 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [RFC v1 proxmox-backup] fix: #3847 pipe from STDIN to proxmox-backup-client X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Hi, thanks for having a look at this issue, this can be an interesting feature. On 11/21/25 4:41 PM, Robert Obkircher wrote: > It was requested [0] that the backup client should suport reading data > from stdin. This could be used to back up files from an unsupported > remote system via ssh or to store the output of mysqldump/pg_dump > commands. > > This is currently not supported, because writing fixed indices needs > to know the size ahead of time. > > This patch adds experimental support for commands like: > cat "$FILE" | proxmox-backup-client backup data.img:/dev/stdin > proxmox-backup-client backup a.img:<(cmd1) b.img:<(cmd2) > > It uses write_at (pwrite64) instead of mmap, because it seemed easier > to get a prototype this way, but it would of course also be possible > to continue using mmap. > > Before I spend more time on this I wanted to ask for some opinions: > > Does it actually make sense to support this at all? Yes, the requested dumping of contents from e.g. a database into a stream which can then be directly be backed up without storing an temporary file are a valid usecase I think. Adding a support for this justified. > If so, should it be a separate endpoint/writer or unified? Using a different endpoint is not required, in my opinion this might however be approach a bit differently than what your initial draft does. I think it would make sense to have the following options for the proxmox-backup-client, both of which can be implemented independent from each other: - Allow the input stream to be a generic raw byte stream by using the img archive name extension, chunking it using the fixed size chunker and upload this as fixed index to the proxmox backup server. - Allow the user to set the mode based on pxar archive name extension (and check the first bytes of the stream for matching pxar magic number) to pass in a pre-encoded pxar stream. This would then however also require to continuosuly check the consistency of the provided pxar stream, aborting if inconsistencies are detected and is more involved. For both cases the chunking, upload and encryption can still be performed using the pre-existing logic. For the FixedIndexWriter as drafted by your patch, I would however suggest to try to approach this a bit differently. This could maybe be done by defining a trait which clearly defines the required method interfaces first, and then implement this trait for the FixedIndexWriterSized and maybe a FixedIndexWriterUnsized? Both implementations could then reuse as much as possible from the pre-existing logic and allow to not interfer with the already existing implementation. Also, the mmapped logic for the writer with already known size must be maintained. What do you think? _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel