From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 480179BEC0 for ; Tue, 21 Nov 2023 15:32:04 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2FEA9BAE2 for ; Tue, 21 Nov 2023 15:32:04 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 21 Nov 2023 15:32:02 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id BD2BB402F0 for ; Tue, 21 Nov 2023 15:32:02 +0100 (CET) From: Hannes Laimer To: pbs-devel@lists.proxmox.com Date: Tue, 21 Nov 2023 15:31:49 +0100 Message-Id: <20231121143155.370916-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.007 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pbs-devel] [PATCH proxmox-backup v6 0/6] local sync-jobs X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2023 14:32:04 -0000 Add support for local sync. SyncJobs without a remote are considered local, and use a different logic for pulling. In the course of adding the extra pull logic, the pull code was rewritten to basically be source independent. Also cli completion and the UI was updated to allow Remotes in SyncJobs to be optional. v6: thanks @Gabriel and @Thomas for the feedback * disallow sync to same datastore * ui: fix not checked checkbox * ui: don't allow same datastore as source as the target * ui: render empty remote-id's the same way namespaces are * reorder patches, do config/api changes last (so instead of an Option pretending to be a String, we have a String pretending to be an Option, avoiding problems in intermediate patches) v5: thanks @Thomas for the feedback * ui: add radio buttons to make local vs remote more clear * move ui patch to the end + api2 -> api v4: thanks @Wolfgang and @Lukas for the feedback * ui: disable rate limit for local sync jobs * ui: rename `Source Remote` -> `Source PBS` * update SYNC_JOB_WORKER_ID_REGEX, use '-' as remote for local * fix problme with groups not being synced to the correct ns * add check for source == target * moved two changes from patch 3 to patch 1 v3: thanks @Fabian and @Wolfang for the feedback * remove enums for Local/Remote * use traits, pull logic now expects a `dyn PullSource`(or `dyn PullReader`) * add lock to dir for local sync * split refactoring of pull logic and implementation of local pulling into two commits v2: thanks @Fabian for the feedback * make pull logic more source independent Hannes Laimer (6): accept a ref to a HttpClient pull: refactor pulling from a datastore pull: add support for pulling from local datastore manager: add completion for opt. Remote in SyncJob api: make Remote for SyncJob optional ui: add support for optional Remote in SyncJob Cargo.toml | 2 + examples/download-speed.rs | 2 +- pbs-api-types/src/jobs.rs | 9 +- pbs-client/src/backup_reader.rs | 2 +- pbs-datastore/src/read_chunk.rs | 2 +- proxmox-backup-client/src/catalog.rs | 4 +- proxmox-backup-client/src/main.rs | 2 +- proxmox-backup-client/src/mount.rs | 2 +- proxmox-file-restore/src/main.rs | 4 +- src/api2/config/remote.rs | 16 +- src/api2/config/sync.rs | 41 +- src/api2/node/tasks.rs | 3 +- src/api2/pull.rs | 60 +- src/bin/proxmox-backup-manager.rs | 67 +- src/bin/proxmox_backup_debug/diff.rs | 2 +- src/server/email_notifications.rs | 18 +- src/server/pull.rs | 1073 ++++++++++++++++---------- www/Utils.js | 5 + www/config/SyncView.js | 1 + www/form/RemoteTargetSelector.js | 36 +- www/window/SyncJobEdit.js | 73 +- 21 files changed, 934 insertions(+), 490 deletions(-) -- 2.39.2