From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 3A0569D81 for ; Mon, 28 Aug 2023 16:42:37 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2275E1EE3B for ; Mon, 28 Aug 2023 16:42:37 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 28 Aug 2023 16:42:35 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 63C3A47253 for ; Mon, 28 Aug 2023 16:42:35 +0200 (CEST) From: Max Carrara To: pbs-devel@lists.proxmox.com Date: Mon, 28 Aug 2023 16:42:02 +0200 Message-Id: <20230828144204.3591503-1-m.carrara@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.083 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [RFC PATCH proxmox-backup 0/2] Introduce experimental `AsyncExtractor` X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Aug 2023 14:42:37 -0000 This RFC proposes an asynchronous implementation of `pbs_client::pxar::extract::{Extractor, ExtractorIter}`. This `AsyncExtractor` has been remodeled from the ground up while preserving the core extraction logic. Its purpose is to provide fully concurrent extraction of pxar files or streams. It does so by offloading every synchronous / blocking call to a separate worker thread with an internal queue. Extraction tasks are executed sequentially to allow for predictable behaviour. The async extractor is intentionally put into a separate module (located at `pbs_client::pxar::aio`), as a complete refactor of every existing extraction-related piece of code is beyond the scope (and intention) of this RFC. Its public API is nowhere near final, but serves its purpose for the time being. Other functions found within `pbs_client::pxar::extract` are not yet implemented. Questions this RFC intends to resolve: 1. In which situations would the `AsyncExtractor` make sense? In which wouldn't it? 2. Should the sync variant be kept around, sharing a `common` implementation with its async variant? If yes, why? 3. Are there any features that the `AsyncExtractor` lacks? Even though of lesser priority, these questions should also be addressed: 4. Which parts of the `AsyncExtractor` are inadequate and could use improvement? 5. Which traits should the `AsyncExtractor` implement (if any?) (e.g. `tokio_stream`, etc.) Furthermore, due to the nature of async applications requiring a runtime in Rust, the `AsyncExtractor` currently suffers from the runtime's overhead. This difference in performance can be seen when comparing the async version of `pxar` (see patch 2) with its current sync counterpart. In my opinion, this does point towards a common implementation which may be used by either sync or async variant, but I am curious to what others have to say. Let me know what you think! :-) Max Carrara (2): pbs-client: pxar: Add prototype implementation of `AsyncExtractor` pxar-bin: Use async instead of sync extractor Cargo.toml | 1 + pbs-client/Cargo.toml | 1 + pbs-client/src/pxar/aio/dir_stack.rs | 543 +++++++++++++++++++ pbs-client/src/pxar/aio/extract/extractor.rs | 446 +++++++++++++++ pbs-client/src/pxar/aio/extract/mod.rs | 220 ++++++++ pbs-client/src/pxar/aio/extract/raw.rs | 503 +++++++++++++++++ pbs-client/src/pxar/aio/metadata.rs | 412 ++++++++++++++ pbs-client/src/pxar/aio/mod.rs | 11 + pbs-client/src/pxar/aio/worker.rs | 167 ++++++ pbs-client/src/pxar/mod.rs | 1 + pxar-bin/src/main.rs | 91 ++-- 11 files changed, 2352 insertions(+), 44 deletions(-) create mode 100644 pbs-client/src/pxar/aio/dir_stack.rs create mode 100644 pbs-client/src/pxar/aio/extract/extractor.rs create mode 100644 pbs-client/src/pxar/aio/extract/mod.rs create mode 100644 pbs-client/src/pxar/aio/extract/raw.rs create mode 100644 pbs-client/src/pxar/aio/metadata.rs create mode 100644 pbs-client/src/pxar/aio/mod.rs create mode 100644 pbs-client/src/pxar/aio/worker.rs -- 2.39.2