Re: [pbs-devel] [RFC proxmox{, -backup} 0/6] add support for configuring max

[Hannes Laimer <h.laimer@proxmox.com>]

On 11/7/25 11:45, Fabian Grünbichler wrote:
> On September 4, 2025 4:37 pm, Hannes Laimer wrote:
>> This uses POSIX semaphores to control how many backups run at any given
>> time. Semaphores seemed like a fitting choice given the problem, so I
>> added a basic wrapper for named ones to proxmox-sys. Also, the changes
>> needed to add concurrency limits in PBS were minimal.
>>
>> A few questions I still have and would be great to get some feedback on,
>>   - do we want to use semaphores like proposed here? Are there any good
>>     reasons not to?
> 
> they are rather primitive, and their file based nature means extra care
> is required to not accidentally reset or mishandle them. reconfiguring
> is also a bit of a pain, as you need to know the old value for certain
> to do so..
> 
> we already have shared memory based synchronization mechanisms in our
> Rust code, which would give us a lot more features.. e.g., allow things
> like registering queued backup sessions if there are no slots available,
> and notifying the oldest one if a slot becomes available.. this is
> currently used by the config version cache and the rate limiting code.
> 
> or if we want to keep it really primitive, we already have a file-based
> operations tracking that we could extend, although that would probably
> require quite a bit of refactoring to make sense, since we currently do
> a lookup/write-operation registration as part of the backup session
> opening/connection upgrade..
> 

good points, seemed a little too fitting for the problem. But you're 
right, there are better alternatives. Thanks for taking a look! :)

>>   - should we include things like verify/gc/prune? Would an extra sem for
>>     reading make sense(so separate limits for r and w, idk if there's a
>>     use-case)?
>>
>> Nothing UI-wise is included, and for changes to apply the proxy has to
>> be restarted. Not sure if restarting the proxy is "ok-ish" ux wise, I
>> guess not :P But changing it involves re-creating the semaphore, which
>> is fine, just that there won't exist one for a really short time...
> 
> what about reloading? we might still have sessions running in the old
> worker that are now no longer accounted for?
> 

true, new proxy doesn't necessarily mean everything finished...

>>
>>
>> proxmox:
>>
>> Hannes Laimer (2):
>>    sys: add wrapper for POSIX semaphores
>>    pbs-api-types: add concurrency_limit to DataStoreConfig
>>
>>   pbs-api-types/src/datastore.rs |   5 +
>>   proxmox-sys/Cargo.toml         |   1 +
>>   proxmox-sys/src/lib.rs         |   2 +
>>   proxmox-sys/src/semaphore.rs   | 164 +++++++++++++++++++++++++++++++++
>>   4 files changed, 172 insertions(+)
>>   create mode 100644 proxmox-sys/src/semaphore.rs
>>
>>
>> proxmox-backup:
>>
>> Hannes Laimer (4):
>>    api: config: update/delete concurrency_limit on datastore
>>    Cargo.toml: add 'semaphore' feature to proxmox-sys dep
>>    bin: proxy: initialize concurrency semaphores for datastores
>>    api: backup: wait for semaphore if one exists
>>
>>   Cargo.toml                      |  2 +-
>>   src/api2/backup/mod.rs          | 24 ++++++++++++++++++++++--
>>   src/api2/config/datastore.rs    |  9 +++++++++
>>   src/bin/proxmox-backup-proxy.rs | 11 +++++++++++
>>   4 files changed, 43 insertions(+), 3 deletions(-)
>>
>>
>> Summary over all repositories:
>>    8 files changed, 215 insertions(+), 3 deletions(-)
>>
>> -- 
>> Generated by git-murpp 0.8.1
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel