From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 5399BC8BC for ; Tue, 12 Apr 2022 07:41:09 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 401DCAB48 for ; Tue, 12 Apr 2022 07:40:39 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 64370AB3D for ; Tue, 12 Apr 2022 07:40:38 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 2F0E940BD1 for ; Tue, 12 Apr 2022 07:40:38 +0200 (CEST) Message-ID: <0f1b92fa-224a-2ba8-820f-a2b3d9731fe3@proxmox.com> Date: Tue, 12 Apr 2022 07:40:35 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Content-Language: en-US To: Proxmox Backup Server development discussion , Hannes Laimer References: <20220412052601.4709-1-h.laimer@proxmox.com> From: Dylan Whyte In-Reply-To: <20220412052601.4709-1-h.laimer@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 1.124 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -1.622 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [restore.rs, maintenance.rs, datastore.rs, pull.rs, mod.rs, status.rs, proxmox-backup-proxy.rs, proxmox-backup-api.rs, lib.rs, backup.rs] Subject: Re: [pbs-devel] [PATCH proxmox-backup v10 0/6] closes #3071: maintenance mode for datastore X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Apr 2022 05:41:09 -0000 Hi, I tested most datastore operations, as well as cases such as enabling maintenance mode mid task, and everything seems to work as expected, except for one potential issue. If you enable maintenance mode during a vzdump backup job, the backup job will complete for the current guest, however, the task log 'client.blob.log' won't get uploaded to the otherwise successful snapshot due to the server being in maintenance mode (this is noted in the PVE task log). Without looking into it, I imagine that this could be somewhat awkward to fix. However, it's also probably not much of an issue, as the task log still exists on the PVE side. I just wanted to make it known. Tested-by: Dylan Whyte On 4/12/22 07:25, Hannes Laimer wrote: > Adds maintenance mode and tracking of active reading/writing operations. > The maintenance mode prevents the start of new operations if the type of > operation they would perform on the datastore would conflict with the > maintenance type that is currently set. This check is performed when > lookup_datastore is called. Tasks only call this function once at the > beginning, therefore updating the maintenance type cannot interfere with > already running tasks. > > active operations tracking: > Changed file layout to now also keep track of the pid+starttime and the > counts of operations that that pid started, like this it is possible to > not count operations that were started by a dead process, since they > are also not active anymore. Whenever the file is updated, also entries > of dead processes are removed. When the file is read, only entries of > active processes are counted. > > The UI shows a spinner with the count of conflictintg tasks (the tasks > that were started before the maintenance type was updated) next to it. > As soon as all conflicting tasks are finished a checkmark appears. > > v10: > - rebase onto master > - minor changes suggested by Dylan on v9 > > v9: > - bump proxmox-schema dep to 1.2.1 > - loosen MAINTENANCE_MESSAGE_REGEX > - MaintenanceMode fn check: use and_then() > - ui: properly (un)escape double quotes in message > > v8: > - replace enum with MaintenanceMode struct > - impl check function on MaintenanceMode struct > - api now encodes type+msg in one string > - message has to be encoded to ensure proper parsing > (everywhere except the UI the message has to be already encoded > when passed, API+CLI) > > v7: > - tracking: fix counting on clone > - ui: correct gettext usages + remove usage of capitalize > > v6: > - also use process start time in order to avoid pid clashes(as suggested > by Thomas and somehow missed by me in the last version) > - now a single call of get_active_operations return reads and writes > - improved code structure > - don't lock when reading > > v5: > - use simple struct and serde instead of manual parsing for file > - move tracking related stuff into new file (task_tracking.rs) > > v4: > - clones are not also tracked > - use lockfile, instead of locking the file > - track pid of the process which started smth > - updating maintenance mode is now always possible > - add get_active_operations endpoint for datastore > - ui: show count of conflicting tasks (or checkmark if no conflicting > operations are active) > > v3, based on Dominik Csapak 's feedback: > - added Operation enum(r/w), as suggested by > - added active operation tracking > - combine type and message into on field > > v2: > - check for maintenance now directly in lookup_datastore > - parameter for checking is now the last acceptable maintenance type, > description in commit msg of 2nd patch > - ui cleanup > > Hannes Laimer (6): > api-types: add maintenance type > datastore: add check for maintenance in lookup > pbs-datastore: add active operations tracking > api: make maintenance_type updatable > api: add get_active_operations endpoint > ui: add option to change the maintenance type > > pbs-api-types/Cargo.toml | 2 +- > pbs-api-types/src/datastore.rs | 23 ++++- > pbs-api-types/src/lib.rs | 3 + > pbs-api-types/src/maintenance.rs | 78 +++++++++++++++++ > pbs-datastore/Cargo.toml | 1 + > pbs-datastore/src/datastore.rs | 126 +++++++++++++++++++-------- > pbs-datastore/src/lib.rs | 4 + > pbs-datastore/src/snapshot_reader.rs | 6 +- > pbs-datastore/src/task_tracking.rs | 110 +++++++++++++++++++++++ > src/api2/admin/datastore.rs | 81 +++++++++++------ > src/api2/backup/mod.rs | 4 +- > src/api2/config/datastore.rs | 5 ++ > src/api2/reader/mod.rs | 6 +- > src/api2/status.rs | 4 +- > src/api2/tape/backup.rs | 6 +- > src/api2/tape/restore.rs | 6 +- > src/bin/proxmox-backup-api.rs | 1 + > src/bin/proxmox-backup-proxy.rs | 4 +- > src/server/mod.rs | 16 +++- > src/server/prune_job.rs | 4 +- > src/server/pull.rs | 4 +- > src/server/verify_job.rs | 4 +- > www/Makefile | 1 + > www/Utils.js | 23 +++++ > www/datastore/OptionView.js | 30 +++++++ > www/window/MaintenanceOptions.js | 77 ++++++++++++++++ > 26 files changed, 539 insertions(+), 90 deletions(-) > create mode 100644 pbs-api-types/src/maintenance.rs > create mode 100644 pbs-datastore/src/task_tracking.rs > create mode 100644 www/window/MaintenanceOptions.js >