[PATCH proxmox-backup] datastore: don't skip empty namespaces on s3 refresh

[PATCH proxmox-backup] datastore: don't skip empty namespaces on s3 refresh

[Hannes Laimer]

Without this empty namespaces present on s3 won't be visible locally.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-datastore/src/datastore.rs | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 7ad3d917..4749319a 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -2535,6 +2535,15 @@ impl DataStore {
                     format!("failed to strip store context prefix {store_prefix} for {object_key}")
                 })?;
                 if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
+                    // Ensure the marker's parent directory exists so the namespace is visible locally
+                    let ns_dir = tmp_base.join(object_path);
+                    if let Some(parent) = ns_dir.parent() {
+                        proxmox_sys::fs::create_path(
+                            parent,
+                            Some(dir_create_options),
+                            Some(dir_create_options),
+                        )?;
+                    }
                     continue;
                 }
 
-- 
2.47.3

Re: [PATCH proxmox-backup] datastore: don't skip empty namespaces on s3 refresh

[Christian Ebner]

Thanks for fixing this issue!

On 3/4/26 11:39 AM, Hannes Laimer wrote:
> Without this empty namespaces present on s3 won't be visible locally.
> 
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>

Tested by creating an (nested as well as root) empty namespace for a 
datastore backed by s3 and performing an s3 refresh. With the patch 
applied the namespace folder structure is created with ownership and 
permissions as expected.

Non empty namespaces did not suffer the issue, but still work as 
expected as well.

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>

Re: [PATCH proxmox-backup] datastore: don't skip empty namespaces on s3 refresh

[Fabian Grünbichler]

On March 4, 2026 11:39 am, Hannes Laimer wrote:
> Without this empty namespaces present on s3 won't be visible locally.
> 
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
>  pbs-datastore/src/datastore.rs | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 7ad3d917..4749319a 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -2535,6 +2535,15 @@ impl DataStore {
>                      format!("failed to strip store context prefix {store_prefix} for {object_key}")
>                  })?;
>                  if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
> +                    // Ensure the marker's parent directory exists so the namespace is visible locally
> +                    let ns_dir = tmp_base.join(object_path);
> +                    if let Some(parent) = ns_dir.parent() {
> +                        proxmox_sys::fs::create_path(
> +                            parent,
> +                            Some(dir_create_options),
> +                            Some(dir_create_options),
> +                        )?;
> +                    }

this is exactly the same code that is done a few lines below (modulo
naming of the variable), could we just move this if with the continue
and the `info!` statement below it a few lines down and have the same
effect?

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 7ad3d917d..ce09a72f1 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -2534,21 +2534,23 @@ impl DataStore {
                 let object_path = object_path.strip_prefix(&store_prefix).with_context(|| {
                     format!("failed to strip store context prefix {store_prefix} for {object_key}")
                 })?;
-                if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
-                    continue;
-                }
-
-                info!("Fetching object {object_path}");
 
                 let file_path = tmp_base.join(object_path);
                 if let Some(parent) = file_path.parent() {
                     proxmox_sys::fs::create_path(
                         parent,
                         Some(dir_create_options),
                         Some(dir_create_options),
                     )?;
                 }
 
+                if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
+                    // namespace created above even if empty
+                    continue;
+                }
+
+                info!("Fetching object {object_path}");
+
                 let mut target_file = tokio::fs::OpenOptions::new()
                     .write(true)
                     .create(true)


>                      continue;
>                  }
>  
> -- 
> 2.47.3
> 
> 
> 
> 
> 
> 

Re: [PATCH proxmox-backup] datastore: don't skip empty namespaces on s3 refresh

[Hannes Laimer]

On 2026-03-05 09:59, Fabian Grünbichler wrote:
> On March 4, 2026 11:39 am, Hannes Laimer wrote:
>> Without this empty namespaces present on s3 won't be visible locally.
>>
>> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
>> ---
>>  pbs-datastore/src/datastore.rs | 9 +++++++++
>>  1 file changed, 9 insertions(+)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 7ad3d917..4749319a 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -2535,6 +2535,15 @@ impl DataStore {
>>                      format!("failed to strip store context prefix {store_prefix} for {object_key}")
>>                  })?;
>>                  if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
>> +                    // Ensure the marker's parent directory exists so the namespace is visible locally
>> +                    let ns_dir = tmp_base.join(object_path);
>> +                    if let Some(parent) = ns_dir.parent() {
>> +                        proxmox_sys::fs::create_path(
>> +                            parent,
>> +                            Some(dir_create_options),
>> +                            Some(dir_create_options),
>> +                        )?;
>> +                    }
> 
> this is exactly the same code that is done a few lines below (modulo
> naming of the variable), could we just move this if with the continue
> and the `info!` statement below it a few lines down and have the same
> effect?
> 

yes, I kind of though there might be some value in stating explicitly
"this is what we do for namespaces" and "this for everything else". But
tbh since these are the exact same there is no point in doing that, I'll
send a v2, thanks!

> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 7ad3d917d..ce09a72f1 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -2534,21 +2534,23 @@ impl DataStore {
>                  let object_path = object_path.strip_prefix(&store_prefix).with_context(|| {
>                      format!("failed to strip store context prefix {store_prefix} for {object_key}")
>                  })?;
> -                if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
> -                    continue;
> -                }
> -
> -                info!("Fetching object {object_path}");
>  
>                  let file_path = tmp_base.join(object_path);
>                  if let Some(parent) = file_path.parent() {
>                      proxmox_sys::fs::create_path(
>                          parent,
>                          Some(dir_create_options),
>                          Some(dir_create_options),
>                      )?;
>                  }
>  
> +                if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
> +                    // namespace created above even if empty
> +                    continue;
> +                }
> +
> +                info!("Fetching object {object_path}");
> +
>                  let mut target_file = tokio::fs::OpenOptions::new()
>                      .write(true)
>                      .create(true)
> 
> 
>>                      continue;
>>                  }
>>  
>> -- 
>> 2.47.3
>>
>>
>>
>>
>>
>>

superseded: [PATCH proxmox-backup] datastore: don't skip empty namespaces on s3 refresh

[Hannes Laimer]

superseded-by:
https://lore.proxmox.com/pbs-devel/20260305092032.29058-1-h.laimer@proxmox.com/T/#u

On 2026-03-04 11:39, Hannes Laimer wrote:
> Without this empty namespaces present on s3 won't be visible locally.
> 
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
>  pbs-datastore/src/datastore.rs | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 7ad3d917..4749319a 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -2535,6 +2535,15 @@ impl DataStore {
>                      format!("failed to strip store context prefix {store_prefix} for {object_key}")
>                  })?;
>                  if object_path.ends_with(NAMESPACE_MARKER_FILENAME) {
> +                    // Ensure the marker's parent directory exists so the namespace is visible locally
> +                    let ns_dir = tmp_base.join(object_path);
> +                    if let Some(parent) = ns_dir.parent() {
> +                        proxmox_sys::fs::create_path(
> +                            parent,
> +                            Some(dir_create_options),
> +                            Some(dir_create_options),
> +                        )?;
> +                    }
>                      continue;
>                  }
>