From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 47EAF79FE8 for ; Thu, 28 Oct 2021 15:01:40 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3EFF720D10 for ; Thu, 28 Oct 2021 15:01:10 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id C554A20D02 for ; Thu, 28 Oct 2021 15:01:06 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 9560945F6E for ; Thu, 28 Oct 2021 15:01:06 +0200 (CEST) From: =?UTF-8?q?Fabian=20Gr=C3=BCnbichler?= To: pbs-devel@lists.proxmox.com Date: Thu, 28 Oct 2021 15:00:46 +0200 Message-Id: <20211028130058.1308810-1-f.gruenbichler@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.341 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [jobs.rs, sync.rs, schema.rs, datastore.rs, proxmox-backup-manager.rs, pull.rs, remote.rs] Subject: [pbs-devel] [PATCH v3 proxmox-backup 0/12] pull/sync group filter X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Oct 2021 13:01:40 -0000 this has been requested a few times on the forum, e.g. for a special sync job for the most important groups, or seeding of a new datastore with a partial view of an existing one. while it's possible to achieve similar results with hacky workarounds based on group ownership and reduced "visibility", implementing it properly is not that complex. possible future additions in a similar fashion: - exclude filters - filtering in other API calls (tape, listing groups/snapshots) - only sync/pull encrypted snapshots (less trusted off-site location) - only sync/pull latest snapshot in each group (fast seeding of new datastore) not changed: BackupGroup was not pulled into api-types - we can still re-evaluate this at some point if we want to switch the split api parameters we currently have, but at the moment this would mean lots of churn for a very small gain. changed since v2: - implement UpdaterType for Vec in proxmox-schema, simplify proxmox-backup patches - pull parameters into struct - added patch for flipped remove-vanished default changed since v1: - reworked filter to support different types, rebased - dropped last patch - add docs patch proxmox: Fabian Grünbichler (1): updater: impl UpdaterType for Vec proxmox-schema/src/schema.rs | 5 +++++ 1 file changed, 5 insertions(+) proxmox-backup: Fabian Grünbichler (11): api-types: add schema for backup group api: add GroupFilter(List) type BackupGroup: add filter helper pull: use BackupGroup consistently pull/sync: extract passed along vars into struct pull: allow pulling groups selectively sync: add group filtering remote: add backup group scanning manager: render group filter properly docs: mention group filter in sync docs fix #sync.cfg/pull: don't remove by default docs/managing-remotes.rst | 6 + debian/postinst | 31 ++++++ pbs-api-types/src/datastore.rs | 5 + pbs-api-types/src/jobs.rs | 64 ++++++++++- pbs-datastore/src/backup_info.rs | 13 +++ src/api2/config/remote.rs | 77 ++++++++++++- src/api2/config/sync.rs | 5 + src/api2/pull.rs | 70 ++++++------ src/bin/proxmox-backup-manager.rs | 117 +++++++++++++++++-- src/bin/proxmox_backup_manager/sync.rs | 21 ++++ src/server/pull.rs | 148 ++++++++++++++++++------- www/config/SyncView.js | 13 ++- www/window/SyncJobEdit.js | 12 ++ 13 files changed, 487 insertions(+), 95 deletions(-) -- 2.30.2 -- 2.30.2