From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 008B081017 for ; Mon, 22 Nov 2021 08:28:15 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E6EA319B28 for ; Mon, 22 Nov 2021 08:28:15 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 1BE6719B17 for ; Mon, 22 Nov 2021 08:28:14 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id DB3BB43DE1 for ; Mon, 22 Nov 2021 08:28:07 +0100 (CET) Message-ID: Date: Mon, 22 Nov 2021 08:28:06 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:95.0) Gecko/20100101 Thunderbird/95.0 Content-Language: en-US To: Proxmox Backup Server development discussion , Hannes Laimer References: <20211112123019.15384-1-h.laimer@proxmox.com> From: Thomas Lamprecht In-Reply-To: <20211112123019.15384-1-h.laimer@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 1.620 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -3.022 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH v4 proxmox-backup 0/6] closes #3071: maintenance mode for datastore X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2021 07:28:16 -0000 On 12.11.21 13:30, Hannes Laimer wrote: > Adds maintenance mode and tracking of active reading/writing operations. > The maintenance mode prevents the start of new operations if the type of > operation they would perform on the datastore would conflict with the > maintenance type that is currently set. This check is performed when > lookup_datastore is called. Tasks only call this function once at the > beginning, therefore updating the maintenance type cannot interfere with > already running tasks. > > active operations tracking: > Changed file layout to now also keep track of the pid and the counts of > operations that that pid started, like this it is possible to not count > operations that were started by a dead process, since they are also not > active anymore. Whenever the file is updated, also entries of dead note that PIDs can be reused, normally one also safes the processes start-time to check for (that's theoretically also racy, but in practice no one can spin that many processes up to force reuse in such a short time). FWIW, your use case here would actually just require the processes start time, you can then simply check if it's older than the one of the current process, if it is, it'd be out of date. > processes are removed. When the file is read, only entries of active > processes are counted. > > The UI shows a spinner with the count of conflictintg tasks (the tasks > that were started before the maintenance type was updated) next to it. > As soon as all conflicting tasks are finished a checkmark appears. > It did not applied cleanly anymore, so I resolved the merge conflicts, fixed a compile error (operations.clone() cannot work as is) applied Dominik's fixes and cleanups, and did another quick round of cleanup myself, reducing indentation depth and line count quite a bit, pushed that out to my internal staff repository as: hannes/datastore-maintenance-v4-with-cleanups Anyhow, the main reason I did not went for this now is the manual file format parser, with serde and now also the shared-memory stuff from Dietmar available we have way better approaches that keeps us more flexible and normally gets a more efficient parser plus more friendly working with the data as programmer. So, please: * check out my branch (you can squash stuff in, at least mine, if you want) the most important thing is that you ensure Dominik's patch that fixes tracking for tape is not lost, as without that it won't work as you was reminded twice by Dominik, IIRC. * create an actual struct for the active operations tracking and either - derive serialize and de-serialize and just save/load it as json - use the shared memory infrastructure, it should cover your use case relatively fine and may be slightly cheaper (as we save the parse step completely) * see if we can move that tracking related stuff in its own file, datastore.rs is already relatively big.. As said, most of it was ok-ish as is and could have been followed up, but I just do not want another self-rolled format, that is hard to expand nicely, out there, especially not as the rust infrastructure we have just makes it so simply to do it in a nicer way.