Re: [pbs-devel] [PATCH proxmox{, -backup} v7 00/47] fix #2943: S3 storage backend for datastores

From: Christian Ebner <c.ebner@proxmox.com>
To: Lukas Wagner <l.wagner@proxmox.com>,
	Proxmox Backup Server development discussion
	<pbs-devel@lists.proxmox.com>
Subject: Re: [pbs-devel] [PATCH proxmox{, -backup} v7 00/47] fix #2943: S3 storage backend for datastores
Date: Tue, 15 Jul 2025 09:32:26 +0200	[thread overview]
Message-ID: <22743186-113e-4edc-87f1-dbe95e0198bb@proxmox.com> (raw)
In-Reply-To: <2bb137c6-f282-4303-bbea-d38f6884ed8b@proxmox.com>

On 7/15/25 09:21, Lukas Wagner wrote:
> 
> 
> On  2025-07-14 17:40, Christian Ebner wrote:
>>
>> Okay, will have a go at this tomorrow and see if I manage to adapt this as well. I agree that reusing the "reuse existing datastore" flag and an automatic s3-refresh might be more intuitive here.
>>
>>>       In the long-term it could be nice be to  actually try to list the contents of
>>>       a bucket and use some heuristics to "find" existing datastores in the bucket
>>>       (could be as easy as trying to find some key that contains ".chunks" in the
>>>       second level, e.g. (somestore/.chunks/...)
>>>       and showing them in some drop-down in the dialog.
>>
>> Keeping this in mind, but this is out of scope for this series, I would rather focus on consolidating the current patches for now.
> 
> Sure, I was just thinking out aloud here, this is of course not required for this series.
> 
>>
>>>       Keeping the use case of 'reusing' an S3 bucket in mind, maybe it would make
>>>       sense to mark 'ownership' of a datastore in the bucket, e.g. in some special marker
>>>       object (could contain the host name, host key fingerprint, machine-id, etc.),
>>>       as to make it harder to accidentally use the same datastore from multiple PBS servers.
>>>       There could be an "export" mechanism, effectively giving up the ownership by clearing
>>>       the marker, signalling it to be safe to re-add it to another PBS server.
>>>       Just capturing some thoughts here. :)
>>
>> Hmm, will keep this in mind as well, although I do not see the benefit of storing the ownership per-se.
>>
>> Ownership and permissions on the bucket and sub-object are best handled by the provider and their acls on tokens.
>>
>> But adding a marker which flags the store as in use seems a good idea and I will see if it makes sense to add this already. If the user wants to reuse a datastore for a PBS instance which is no longer available or failed, removing the marker by some other means (e.g. provider tooling) first should be acceptable as fail safe I think.
> 
> Yeah, ultimately that's what I meant with 'ownership' - marking the datastore as in use. Using some unique
> identifier as a marker could then also show 'used by whom?' (as in, which PBS host uses it -> ownership).

True, might make sense to store this as json file then so it can be 
easily expanded in the future if the need for further information about 
the PBS instance and datastore need to be included.

> 
> I guess it would be quite easy to implement a 'force' flag when importing a datastore, which will just
> ignore the 'in use'/'owner' flag, replacing the flag with its own. This would then be, as you said,
> useful for then the original PBS cannot give up the flag because it failed/is no longer available.
Okay, sounds good to me, will see to implement that as well then.

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel