From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id B930C70D1A for ; Wed, 15 Jun 2022 10:21:14 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id AFFEE27689 for ; Wed, 15 Jun 2022 10:20:44 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id AEE692767F for ; Wed, 15 Jun 2022 10:20:43 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 7ECF643B4A for ; Wed, 15 Jun 2022 10:20:43 +0200 (CEST) From: Stefan Sterz To: pbs-devel@lists.proxmox.com Date: Wed, 15 Jun 2022 10:20:36 +0200 Message-Id: <20220615082040.96959-1-s.sterz@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.058 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [sync.rs, pull.rs, jobs.rs, proxmox-backup-manager.rs] Subject: [pbs-devel] [RFC PATCH 0/4] fix #3786: add a "deep sync" option to sync jobs X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2022 08:21:14 -0000 this series adds a "deep sync" option to sync jobs. a deep sync uses the information from a previous verification job to re-sync snapshots that have corrupted chunks. the deep sync option is added to the "advanced" section of the sync job configuration intentionally for the reasoning see commit 3. however, i am not entirely sure if that's the best way of handling this. sending this as an rfc because: a) this is a fairly minimalistic and unoptimized implementation. a deep sync currently just looks at _every_ manifest and if it's corrupt re-syncs the snapshot. as described in the bug report this could be improved by flagging groups during gc / verification / etc jobs and only looking at groups with corrupt snapshots. however, this would require a place to store this information. afaict currently the only metadata we have about groups is their owner and that has a dedicated "owner" file. i could just add a "corrupted" file, but if we need further information about groups in the future it might make sense to think about a more flexible solution. hence, i see three options: a) stick with this simple implementation, b) leverage gc / verification / etc jobs further and flag groups with a dedicated "corrupt" file or c) add a yet to be determined meta-data file for groups and store the flag there. any input would be greatly appreciated. b) while i tested this code quite a bit, i am unsure whether i missed scenarios in which corrupted snapshots might not be re-synced. Stefan Sterz (4): fix #3786: api: add deep sync parameter fix #3786: server/datastore: add deep sync parameter to pull sync jobs fix #3786: ui/cli: add deep sync option to ui and cli fix #3786: docs: document deep sync behavior and prerequisites docs/managing-remotes.rst | 6 ++++++ pbs-api-types/src/jobs.rs | 12 ++++++++++++ pbs-datastore/src/backup_info.rs | 22 +++++++++++++++++++++- src/api2/config/sync.rs | 10 ++++++++++ src/api2/pull.rs | 12 ++++++++++-- src/bin/proxmox-backup-manager.rs | 13 +++++++++++-- src/server/pull.rs | 29 +++++++++++++++++++++-------- www/window/SyncJobEdit.js | 11 +++++++++++ 8 files changed, 102 insertions(+), 13 deletions(-) -- 2.30.2