From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <h.laimer@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id 92BA8F573
 for <pbs-devel@lists.proxmox.com>; Fri, 29 Sep 2023 14:49:15 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id 72FA188E
 for <pbs-devel@lists.proxmox.com>; Fri, 29 Sep 2023 14:49:15 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
 [94.136.29.106])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS
 for <pbs-devel@lists.proxmox.com>; Fri, 29 Sep 2023 14:49:14 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
 by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 6AC6E4874D
 for <pbs-devel@lists.proxmox.com>; Fri, 29 Sep 2023 14:49:14 +0200 (CEST)
From: Hannes Laimer <h.laimer@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Fri, 29 Sep 2023 14:48:55 +0200
Message-Id: <20230929124901.179405-1-h.laimer@proxmox.com>
X-Mailer: git-send-email 2.39.2
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results:  0
 AWL -0.013 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DMARC_MISSING             0.1 Missing DMARC policy
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
 URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
 information. [diff.rs, main.rs, catalog.rs, remote.rs, download-speed.rs,
 jobs.rs, sync.rs, pull.rs, mount.rs, proxmox-backup-manager.rs, tasks.rs]
Subject: [pbs-devel] [PATCH proxmox-backup v4 0/6] local sync-jobs
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Fri, 29 Sep 2023 12:49:15 -0000

Add support for local sync. SyncJobs without a remote are considered local, and
use a different logic for pulling. In the course of adding the extra pull logic,
the pull code was rewritten to basically be source independent. Also cli
completion and the UI was updated to allow Remotes in SyncJobs to be optional.

v2: thanks @Fabian for the feedback
 * make pull logic more source independent

v3: thanks @Fabian and @Wolfang for the feedback
 * remove enums for Local/Remote
 * use traits, pull logic now expects a `dyn PullSource`(or `dyn PullReader`)
 * add lock to dir for local sync
 * split refactoring of pull logic and implementation of
    local pulling into two commits

v4: thanks @Wolfgang and @Lukas for the feedback
 * ui: disable rate limit for local sync jobs
 * ui: rename `Source Remote` -> `Source PBS`
 * update SYNC_JOB_WORKER_ID_REGEX, use '-' as remote for local
 * fix problme with groups not being synced to the correct ns
 * add check for source == target
 * moved two changes from patch 3 to patch 1

Hannes Laimer (6):
  api2: make Remote for SyncJob optional
  ui: add support for optional Remote in SyncJob
  manager: add completion for opt. Remote in SyncJob
  accept a ref to a HttpClient
  pull: refactor pulling from a datastore
  pull: add support for pulling from local datastore

 Cargo.toml                           |    2 +
 examples/download-speed.rs           |    2 +-
 pbs-api-types/src/jobs.rs            |    9 +-
 pbs-client/src/backup_reader.rs      |    2 +-
 pbs-datastore/src/read_chunk.rs      |    2 +-
 proxmox-backup-client/src/catalog.rs |    4 +-
 proxmox-backup-client/src/main.rs    |    2 +-
 proxmox-backup-client/src/mount.rs   |    2 +-
 proxmox-file-restore/src/main.rs     |    4 +-
 src/api2/config/remote.rs            |   16 +-
 src/api2/config/sync.rs              |   41 +-
 src/api2/node/tasks.rs               |    3 +-
 src/api2/pull.rs                     |   59 +-
 src/bin/proxmox-backup-manager.rs    |   67 +-
 src/bin/proxmox_backup_debug/diff.rs |    2 +-
 src/server/email_notifications.rs    |   18 +-
 src/server/pull.rs                   | 1073 ++++++++++++++++----------
 www/form/RemoteTargetSelector.js     |   29 +-
 www/window/SyncJobEdit.js            |   15 +-
 19 files changed, 860 insertions(+), 492 deletions(-)

-- 
2.39.2