Re: [pbs-devel] [PATCH proxmox{, -backup} v7 00/47] fix #2943: S3 storage backend for datastores

From: Lukas Wagner <l.wagner@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>,
	Proxmox Backup Server development discussion
	<pbs-devel@lists.proxmox.com>
Subject: Re: [pbs-devel] [PATCH proxmox{, -backup} v7 00/47] fix #2943: S3 storage backend for datastores
Date: Tue, 15 Jul 2025 09:21:13 +0200	[thread overview]
Message-ID: <2bb137c6-f282-4303-bbea-d38f6884ed8b@proxmox.com> (raw)
In-Reply-To: <208b3247-cc11-46b0-8e2e-603cbcffe763@proxmox.com>

On  2025-07-14 17:40, Christian Ebner wrote:
> 
> Okay, will have a go at this tomorrow and see if I manage to adapt this as well. I agree that reusing the "reuse existing datastore" flag and an automatic s3-refresh might be more intuitive here.
> 
>>      In the long-term it could be nice be to  actually try to list the contents of
>>      a bucket and use some heuristics to "find" existing datastores in the bucket
>>      (could be as easy as trying to find some key that contains ".chunks" in the
>>      second level, e.g. (somestore/.chunks/...)
>>      and showing them in some drop-down in the dialog.
> 
> Keeping this in mind, but this is out of scope for this series, I would rather focus on consolidating the current patches for now.

Sure, I was just thinking out aloud here, this is of course not required for this series.

> 
>>      Keeping the use case of 'reusing' an S3 bucket in mind, maybe it would make
>>      sense to mark 'ownership' of a datastore in the bucket, e.g. in some special marker
>>      object (could contain the host name, host key fingerprint, machine-id, etc.),
>>      as to make it harder to accidentally use the same datastore from multiple PBS servers.
>>      There could be an "export" mechanism, effectively giving up the ownership by clearing
>>      the marker, signalling it to be safe to re-add it to another PBS server.
>>      Just capturing some thoughts here. :)
> 
> Hmm, will keep this in mind as well, although I do not see the benefit of storing the ownership per-se.
> 
> Ownership and permissions on the bucket and sub-object are best handled by the provider and their acls on tokens.
> 
> But adding a marker which flags the store as in use seems a good idea and I will see if it makes sense to add this already. If the user wants to reuse a datastore for a PBS instance which is no longer available or failed, removing the marker by some other means (e.g. provider tooling) first should be acceptable as fail safe I think.

Yeah, ultimately that's what I meant with 'ownership' - marking the datastore as in use. Using some unique
identifier as a marker could then also show 'used by whom?' (as in, which PBS host uses it -> ownership).

I guess it would be quite easy to implement a 'force' flag when importing a datastore, which will just
ignore the 'in use'/'owner' flag, replacing the flag with its own. This would then be, as you said,
useful for then the original PBS cannot give up the flag because it failed/is no longer available.

-- 
- Lukas

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel