public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH proxmox-backup] client: catalog: improve error message for missing catalog
@ 2024-07-15 13:46 Christian Ebner
  2024-07-16  8:49 ` Fabian Grünbichler
  0 siblings, 1 reply; 5+ messages in thread
From: Christian Ebner @ 2024-07-15 13:46 UTC (permalink / raw)
  To: pbs-devel

Commit c0302805c "client: backup: conditionally write catalog for
file level backups" drops encoding of the dedicated catalog when
archives are encoded as split metadata/data archives with the
`change-detection-mode` set to `data` or `metadata`.

Since the catalog is not present anymore, accessing these snapshots
via the `proxmox-backup-client catalog` command is obsolete for these
cases, one should opt for the `proxmox-backup-client mount` command
to inspect these archives instead.

Improve the error message one gets when trying to access such
snapshots by checking if the catalog is present in the manifest and if
not, check if it contains a split pxar archive.

A caller now gets the following error message:

```
no catalog for split pxar archives, use `mount` instead to inspect content
```

instead of the rather generic

```
Unable to open dynamic index "/<snapshot-path>/catalog.pcat1.didx"
- No such file or directory (os error 2)
```

Reported in the community forum:
https://forum.proxmox.com/threads/150713/

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-backup-client/src/catalog.rs | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/proxmox-backup-client/src/catalog.rs b/proxmox-backup-client/src/catalog.rs
index 276457c1d..7e3bdd4e9 100644
--- a/proxmox-backup-client/src/catalog.rs
+++ b/proxmox-backup-client/src/catalog.rs
@@ -14,6 +14,7 @@ use pbs_client::tools::key_source::get_encryption_key_password;
 use pbs_client::{BackupReader, RemoteChunkReader};
 use pbs_tools::crypt_config::CryptConfig;
 use pbs_tools::json::required_string_param;
+use pbs_datastore::manifest::{BackupManifest, FileInfo};
 
 use crate::helper;
 use crate::{
@@ -24,6 +25,24 @@ use crate::{
     IndexFile, Shell, CATALOG_NAME, KEYFD_SCHEMA, REPO_URL_SCHEMA,
 };
 
+fn lookup_catalog_file_info(manifest: &BackupManifest) -> Result<&FileInfo, Error> {
+    let file_info = match manifest.lookup_file_info(CATALOG_NAME) {
+        Ok(file_info) => file_info,
+        Err(err) => {
+            for file_info in manifest.files() {
+                if file_info.filename.ends_with(".mpxar.didx") {
+                    bail!(
+                        "no catalog for split pxar archives, use `mount` instead to inspect content"
+                    );
+                }
+            }
+            return Err(err);
+        }
+    };
+
+    Ok(file_info)
+}
+
 #[api(
    input: {
         properties: {
@@ -89,14 +108,14 @@ async fn dump_catalog(param: Value) -> Result<Value, Error> {
     let (manifest, _) = client.download_manifest().await?;
     manifest.check_fingerprint(crypt_config.as_ref().map(Arc::as_ref))?;
 
+    let file_info = lookup_catalog_file_info(&manifest)?;
+
     let index = client
         .download_dynamic_index(&manifest, CATALOG_NAME)
         .await?;
 
     let most_used = index.find_most_used_chunks(8);
 
-    let file_info = manifest.lookup_file_info(CATALOG_NAME)?;
-
     let chunk_reader = RemoteChunkReader::new(
         client.clone(),
         crypt_config,
@@ -207,6 +226,8 @@ async fn catalog_shell(param: Value) -> Result<(), Error> {
     let (manifest, _) = client.download_manifest().await?;
     manifest.check_fingerprint(crypt_config.as_ref().map(Arc::as_ref))?;
 
+    lookup_catalog_file_info(&manifest)?;
+
     let decoder = helper::get_pxar_fuse_accessor(
         &server_archive_name,
         client.clone(),
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [pbs-devel] [PATCH proxmox-backup] client: catalog: improve error message for missing catalog
  2024-07-15 13:46 [pbs-devel] [PATCH proxmox-backup] client: catalog: improve error message for missing catalog Christian Ebner
@ 2024-07-16  8:49 ` Fabian Grünbichler
  2024-07-16  8:54   ` Christian Ebner
  0 siblings, 1 reply; 5+ messages in thread
From: Fabian Grünbichler @ 2024-07-16  8:49 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Christian Ebner


> Christian Ebner <c.ebner@proxmox.com> hat am 15.07.2024 15:46 CEST geschrieben:
> 
>  
> Commit c0302805c "client: backup: conditionally write catalog for
> file level backups" drops encoding of the dedicated catalog when
> archives are encoded as split metadata/data archives with the
> `change-detection-mode` set to `data` or `metadata`.
> 
> Since the catalog is not present anymore, accessing these snapshots
> via the `proxmox-backup-client catalog` command is obsolete for these
> cases, one should opt for the `proxmox-backup-client mount` command
> to inspect these archives instead.
> 
> Improve the error message one gets when trying to access such
> snapshots by checking if the catalog is present in the manifest and if
> not, check if it contains a split pxar archive.
> 
> A caller now gets the following error message:
> 
> ```
> no catalog for split pxar archives, use `mount` instead to inspect content
> ```
> 
> instead of the rather generic
> 
> ```
> Unable to open dynamic index "/<snapshot-path>/catalog.pcat1.didx"
> - No such file or directory (os error 2)
> ```

alternatively, couldn't we adapt the CLI call like we did the API endpoint, and print a file listing using the metadata archive? mounting and then calling find seems a bit roundabout, if we can easily offer the information directly?

> Reported in the community forum:
> https://forum.proxmox.com/threads/150713/
> 
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>  proxmox-backup-client/src/catalog.rs | 25 +++++++++++++++++++++++--
>  1 file changed, 23 insertions(+), 2 deletions(-)
> 
> diff --git a/proxmox-backup-client/src/catalog.rs b/proxmox-backup-client/src/catalog.rs
> index 276457c1d..7e3bdd4e9 100644
> --- a/proxmox-backup-client/src/catalog.rs
> +++ b/proxmox-backup-client/src/catalog.rs
> @@ -14,6 +14,7 @@ use pbs_client::tools::key_source::get_encryption_key_password;
>  use pbs_client::{BackupReader, RemoteChunkReader};
>  use pbs_tools::crypt_config::CryptConfig;
>  use pbs_tools::json::required_string_param;
> +use pbs_datastore::manifest::{BackupManifest, FileInfo};
>  
>  use crate::helper;
>  use crate::{
> @@ -24,6 +25,24 @@ use crate::{
>      IndexFile, Shell, CATALOG_NAME, KEYFD_SCHEMA, REPO_URL_SCHEMA,
>  };
>  
> +fn lookup_catalog_file_info(manifest: &BackupManifest) -> Result<&FileInfo, Error> {
> +    let file_info = match manifest.lookup_file_info(CATALOG_NAME) {
> +        Ok(file_info) => file_info,
> +        Err(err) => {
> +            for file_info in manifest.files() {
> +                if file_info.filename.ends_with(".mpxar.didx") {
> +                    bail!(
> +                        "no catalog for split pxar archives, use `mount` instead to inspect content"
> +                    );
> +                }
> +            }
> +            return Err(err);
> +        }
> +    };
> +
> +    Ok(file_info)
> +}
> +
>  #[api(
>     input: {
>          properties: {
> @@ -89,14 +108,14 @@ async fn dump_catalog(param: Value) -> Result<Value, Error> {
>      let (manifest, _) = client.download_manifest().await?;
>      manifest.check_fingerprint(crypt_config.as_ref().map(Arc::as_ref))?;
>  
> +    let file_info = lookup_catalog_file_info(&manifest)?;
> +
>      let index = client
>          .download_dynamic_index(&manifest, CATALOG_NAME)
>          .await?;
>  
>      let most_used = index.find_most_used_chunks(8);
>  
> -    let file_info = manifest.lookup_file_info(CATALOG_NAME)?;
> -
>      let chunk_reader = RemoteChunkReader::new(
>          client.clone(),
>          crypt_config,
> @@ -207,6 +226,8 @@ async fn catalog_shell(param: Value) -> Result<(), Error> {
>      let (manifest, _) = client.download_manifest().await?;
>      manifest.check_fingerprint(crypt_config.as_ref().map(Arc::as_ref))?;
>  
> +    lookup_catalog_file_info(&manifest)?;
> +
>      let decoder = helper::get_pxar_fuse_accessor(
>          &server_archive_name,
>          client.clone(),
> -- 
> 2.39.2
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [pbs-devel] [PATCH proxmox-backup] client: catalog: improve error message for missing catalog
  2024-07-16  8:49 ` Fabian Grünbichler
@ 2024-07-16  8:54   ` Christian Ebner
  2024-07-16  9:01     ` Fabian Grünbichler
  0 siblings, 1 reply; 5+ messages in thread
From: Christian Ebner @ 2024-07-16  8:54 UTC (permalink / raw)
  To: Fabian Grünbichler, Proxmox Backup Server development discussion

On 7/16/24 10:49, Fabian Grünbichler wrote:
> 
>> Christian Ebner <c.ebner@proxmox.com> hat am 15.07.2024 15:46 CEST geschrieben:
>>
>>   
>> Commit c0302805c "client: backup: conditionally write catalog for
>> file level backups" drops encoding of the dedicated catalog when
>> archives are encoded as split metadata/data archives with the
>> `change-detection-mode` set to `data` or `metadata`.
>>
>> Since the catalog is not present anymore, accessing these snapshots
>> via the `proxmox-backup-client catalog` command is obsolete for these
>> cases, one should opt for the `proxmox-backup-client mount` command
>> to inspect these archives instead.
>>
>> Improve the error message one gets when trying to access such
>> snapshots by checking if the catalog is present in the manifest and if
>> not, check if it contains a split pxar archive.
>>
>> A caller now gets the following error message:
>>
>> ```
>> no catalog for split pxar archives, use `mount` instead to inspect content
>> ```
>>
>> instead of the rather generic
>>
>> ```
>> Unable to open dynamic index "/<snapshot-path>/catalog.pcat1.didx"
>> - No such file or directory (os error 2)
>> ```
> 
> alternatively, couldn't we adapt the CLI call like we did the API endpoint, and print a file listing using the metadata archive? mounting and then calling find seems a bit roundabout, if we can easily offer the information directly?
> 

Yes, for the catalog dump this would indeed be similar to the API 
endpoint. I however did not want to go that route, because then one 
should offer the catalog shell as well I guess?



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [pbs-devel] [PATCH proxmox-backup] client: catalog: improve error message for missing catalog
  2024-07-16  8:54   ` Christian Ebner
@ 2024-07-16  9:01     ` Fabian Grünbichler
  2024-07-16  9:04       ` Christian Ebner
  0 siblings, 1 reply; 5+ messages in thread
From: Fabian Grünbichler @ 2024-07-16  9:01 UTC (permalink / raw)
  To: Christian Ebner, Proxmox Backup Server development discussion


> Christian Ebner <c.ebner@proxmox.com> hat am 16.07.2024 10:54 CEST geschrieben:
> 
>  
> On 7/16/24 10:49, Fabian Grünbichler wrote:
> > 
> >> Christian Ebner <c.ebner@proxmox.com> hat am 15.07.2024 15:46 CEST geschrieben:
> >>
> >>   
> >> Commit c0302805c "client: backup: conditionally write catalog for
> >> file level backups" drops encoding of the dedicated catalog when
> >> archives are encoded as split metadata/data archives with the
> >> `change-detection-mode` set to `data` or `metadata`.
> >>
> >> Since the catalog is not present anymore, accessing these snapshots
> >> via the `proxmox-backup-client catalog` command is obsolete for these
> >> cases, one should opt for the `proxmox-backup-client mount` command
> >> to inspect these archives instead.
> >>
> >> Improve the error message one gets when trying to access such
> >> snapshots by checking if the catalog is present in the manifest and if
> >> not, check if it contains a split pxar archive.
> >>
> >> A caller now gets the following error message:
> >>
> >> ```
> >> no catalog for split pxar archives, use `mount` instead to inspect content
> >> ```
> >>
> >> instead of the rather generic
> >>
> >> ```
> >> Unable to open dynamic index "/<snapshot-path>/catalog.pcat1.didx"
> >> - No such file or directory (os error 2)
> >> ```
> > 
> > alternatively, couldn't we adapt the CLI call like we did the API endpoint, and print a file listing using the metadata archive? mounting and then calling find seems a bit roundabout, if we can easily offer the information directly?
> > 
> 
> Yes, for the catalog dump this would indeed be similar to the API 
> endpoint. I however did not want to go that route, because then one 
> should offer the catalog shell as well I guess?

that should be do-able as well I guess (although the advantage is a bit less clear there, other than muscle-memory/convenience ;))


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [pbs-devel] [PATCH proxmox-backup] client: catalog: improve error message for missing catalog
  2024-07-16  9:01     ` Fabian Grünbichler
@ 2024-07-16  9:04       ` Christian Ebner
  0 siblings, 0 replies; 5+ messages in thread
From: Christian Ebner @ 2024-07-16  9:04 UTC (permalink / raw)
  To: Fabian Grünbichler, Proxmox Backup Server development discussion

On 7/16/24 11:01, Fabian Grünbichler wrote:
> 
>> Christian Ebner <c.ebner@proxmox.com> hat am 16.07.2024 10:54 CEST geschrieben:
>>
>>   
>> On 7/16/24 10:49, Fabian Grünbichler wrote:
>>>
>>>> Christian Ebner <c.ebner@proxmox.com> hat am 15.07.2024 15:46 CEST geschrieben:
>>>>
>>>>    
>>>> Commit c0302805c "client: backup: conditionally write catalog for
>>>> file level backups" drops encoding of the dedicated catalog when
>>>> archives are encoded as split metadata/data archives with the
>>>> `change-detection-mode` set to `data` or `metadata`.
>>>>
>>>> Since the catalog is not present anymore, accessing these snapshots
>>>> via the `proxmox-backup-client catalog` command is obsolete for these
>>>> cases, one should opt for the `proxmox-backup-client mount` command
>>>> to inspect these archives instead.
>>>>
>>>> Improve the error message one gets when trying to access such
>>>> snapshots by checking if the catalog is present in the manifest and if
>>>> not, check if it contains a split pxar archive.
>>>>
>>>> A caller now gets the following error message:
>>>>
>>>> ```
>>>> no catalog for split pxar archives, use `mount` instead to inspect content
>>>> ```
>>>>
>>>> instead of the rather generic
>>>>
>>>> ```
>>>> Unable to open dynamic index "/<snapshot-path>/catalog.pcat1.didx"
>>>> - No such file or directory (os error 2)
>>>> ```
>>>
>>> alternatively, couldn't we adapt the CLI call like we did the API endpoint, and print a file listing using the metadata archive? mounting and then calling find seems a bit roundabout, if we can easily offer the information directly?
>>>
>>
>> Yes, for the catalog dump this would indeed be similar to the API
>> endpoint. I however did not want to go that route, because then one
>> should offer the catalog shell as well I guess?
> 
> that should be do-able as well I guess (although the advantage is a bit less clear there, other than muscle-memory/convenience ;))

Okay, than I will have a go at implementing this!


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2024-07-16  9:04 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-07-15 13:46 [pbs-devel] [PATCH proxmox-backup] client: catalog: improve error message for missing catalog Christian Ebner
2024-07-16  8:49 ` Fabian Grünbichler
2024-07-16  8:54   ` Christian Ebner
2024-07-16  9:01     ` Fabian Grünbichler
2024-07-16  9:04       ` Christian Ebner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal