From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id DDEA41FF15E for ; Fri, 9 Aug 2024 16:20:54 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BF3883F03A; Fri, 9 Aug 2024 16:21:04 +0200 (CEST) Mime-Version: 1.0 Date: Fri, 09 Aug 2024 16:20:30 +0200 Message-Id: To: "Dominik Csapak" , "Proxmox Backup Server development discussion" From: "Max Carrara" X-Mailer: aerc 0.17.0-72-g6a84f1331f1c References: In-Reply-To: X-SPAM-LEVEL: Spam detection results: 0 AWL 0.033 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [docs.rs] Subject: Re: [pbs-devel] RFC: Scheduler for PBS X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" On Fri Aug 9, 2024 at 2:52 PM CEST, Dominik Csapak wrote: > Hi, > > great to see that you tackle this! > > I read through the overview, which sounds fine, but I think that it > should more reflect the actual issues, namely limitations in memory, > threads, disk io and network. > > The actual reason people want to schedule things is to not overload the system > (because of timeouts, hangs, etc.) so any scheduling system should consider > not only the amount of jobs, but how much resources the the job will/can > utilize. > > E.g. when I tried to introduce multi-threaded tape backup (configurable threads > per tape job), Thomas rightfully said that it's probably not a good idea, since > making multiple parallel tape backup job increases the load by much more than before. > > I generally like the approach, but I personally would like to see some > work with resource constraints, for example one could imagine a configurable > amount of available threads and (configurable?) used thread by job type > > so i can set my available to e.g. 10 and if my tape backup jobs then get > 4, i can start 2 in parallel but not more > > Such a system does not have to be included from the beginning IMO, but the > architecture should be prepared for such things > > Does that make sense? That does make sense, yes! Thanks for bringing this to our attention. We've just discussed this off-list a bit and mostly agree on stuff like e.g. the thread limit per worker - though to be sure, do you mean the number of threads that are passed to e.g. a `ParallelHandler` and similar? The scheduler doesn't really have a way to *really* enforce any limits, though with the event-based architecture, it should be fairly trivial to just add new fields to the scheduler's config. We want to have a kind of "top-down control", so once the scheduler can actually spawn and manage tasks itself (not like how it's done right now, see my response to Chris), the scheduler could give the task a separate thread pool for the stuff it wants to run in parallel. There could even be different "types" of thread pools depending on the purpose. This is much easier said than done though, but I'm honestly rather confident that we can get this to work. I would prefer to have the resource-checking and -management decoupled and warded off, so that the scheduler itself isn't really concerned with that. Rather, it should ask the (e.g.) `ResourceManager` if there are enough threads available for a `JobType::TapeBackup` or something of the sort. Another thing we've been discussing just now was to just give the spawned task a struct representing the limits it should abide to - that would be a soft limit, but it would make things probably a lot easier. (After all, passing a thread pool to the task also doesn't mean the task *has* to use that thread pool...) One thing I just discovered is tokio's `Semaphore` [1], which we could use to keep track of the resources we've been handing out. So, IMO this is a good idea and something we definitely should consider in the future, though I have a couple questions: 1. How would you track & enforce memory limits? I think this is a much harder problem, to be honest. 2. In the same vein, how could one find out how much memory a given task will use? There's nothing that prevents tasks from just allocating more memory at will, obviously. Do you rather mean that if there's e.g. >90% memory being used (can be made configurable), that we're not spawning any additional tasks? 3. How would you limit disk IO? We definitely want to add a limit for the number of jobs that can run on a datastore at a time, so I guess that would also be indirectly included there..? (It could probably also be done with tokio's `Semaphore` [1], but we'd need some kind of abstraction on top of that, because we can still just read / write / open / close at will etc. We would need a uniform way of accessing disk resources and *not* use any other way to perform disk IO otherwise, which will be *hard*) 4. I guess network limits (e.g. bandwidth limits for sync jobs etc.) could just be enforced on the TCP socket, so this shouldn't be too hard. That way you could enforce individual rate limits for individual tasks. Though, probably also easier said than done. Can you elaborate some more on this, too? Thanks a lot for your input, you've given us lots of ideas as well! :) [1]: https://docs.rs/tokio/latest/tokio/sync/struct.Semaphore.html _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel