[PATCH proxmox-backup] fix #5748: docs: add `catalog.pcat1` format specification

[PATCH proxmox-backup] fix #5748: docs: add `catalog.pcat1` format specification

[Shan Shaji]

Earlier proxmox catalog file format specification was not present
inside the docs. fixed it by adding format specification under file
formats.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=5748
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 changes since RFC: Thanks @Christian Ebner
 - Changed from graph to table layout.
 - Explicity mention about the variable length types. 
 - Denoted the root start offset as u64.
 - Add text description for each field.
 
 docs/file-formats.rst | 65 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/docs/file-formats.rst b/docs/file-formats.rst
index ed3250d11..52cb20ee0 100644
--- a/docs/file-formats.rst
+++ b/docs/file-formats.rst
@@ -23,6 +23,71 @@ overhead introduced by the file payloads.
 
 .. graphviz:: meta-format-overview.dot
 
+.. _pcat1-format:
+
+Proxmox Catalog File Format (``.pcat1``)
+----------------------------------------
+
+The asterisk notation indicates variable-length encoding: u64* denotes an unsigned variable-length encoded integer, while i64*
+denotes a signed variable-length encoded integer.
+
+.. list-table::
+   :widths: auto
+
+   * - ``MAGIC: [u8; 8]``
+     - ``[145, 253, 96, 249, 196, 103, 88, 213]``.
+   * - ``TABLE_LENGTH: u64*``
+     - Total length of the contents inside the Directory.
+   * - ``ENTRY_COUNT: u64*``
+     - Count of entries present inside the Directory.
+   * - ``ENTRIES``
+     - Concatinated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout <entry-layout>` .) Repeated ``ENTRY_COUNT`` times.
+   * - ...
+     - Further concatination of sequential directory blocks (Table Length, Entry Count and Entries).
+   * - ``ROOT_START_OFFSET: u64``
+     - Root marker (/) start offset. Stored as **little-endian**
+
+.. _entry-layout:
+
+Except for files and directories, no additional payload is stored beyond the name and its length inside an ENTRY.
+
+.. list-table::
+   :widths: auto
+
+   * - ``ENTRY_TYPE: u8``
+     - The type of entry (see entry :ref:`types <entry-types>`).
+   * - ``NAME_LENGTH: u64*``
+     - Length of file or directory name. Stored as variable length encoded u64.
+   * - ``NAME: [u8]``
+     - The raw UTF-8 string bytes of the name.
+   * - ``PAYLOAD``
+     - Bytes based on ``ENTRY_TYPE``
+
+       * **Directory ('d')**: ``offset_back: u64*`` (Child-to-Parent offset distance)
+       * **File ('f')**: ``size: u64*`` followed by ``mtime: i64*``
+.. _entry-types:
+
+Entry Types:
+
+.. list-table::
+   :widths: auto
+
+   * - ``d``
+     - Directory
+   * - ``f``
+     - File
+   * - ``l``
+     - Symlink
+   * - ``h``
+     - Hardlink
+   * - ``b``
+     - Block device
+   * - ``c``
+     - Char device
+   * - ``p``
+     - FIFO
+   * - ``s``
+     - Socket
 .. _ppxar-format:
 
 Proxmox File Archive Format - Payload (``.ppxar``)
-- 
2.47.3

Re: [PATCH proxmox-backup] fix #5748: docs: add `catalog.pcat1` format specification

[Fabian Grünbichler]

On July 2, 2026 1:51 pm, Shan Shaji wrote:
> Earlier proxmox catalog file format specification was not present
> inside the docs. fixed it by adding format specification under file
> formats.
> 
> Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=5748
> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
> ---
>  changes since RFC: Thanks @Christian Ebner
>  - Changed from graph to table layout.
>  - Explicity mention about the variable length types. 
>  - Denoted the root start offset as u64.
>  - Add text description for each field.
>  
>  docs/file-formats.rst | 65 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 65 insertions(+)
> 
> diff --git a/docs/file-formats.rst b/docs/file-formats.rst
> index ed3250d11..52cb20ee0 100644
> --- a/docs/file-formats.rst
> +++ b/docs/file-formats.rst
> @@ -23,6 +23,71 @@ overhead introduced by the file payloads.
>  
>  .. graphviz:: meta-format-overview.dot
>  
> +.. _pcat1-format:
> +
> +Proxmox Catalog File Format (``.pcat1``)
> +----------------------------------------

this should go between pxar and mpxar, or below ppxar. there's arguments
for both, but between mpxar and ppxar which always form a pair and never
use catalog files is the wrong place ;)

> +
> +The asterisk notation indicates variable-length encoding: u64* denotes an unsigned variable-length encoded integer, while i64*
> +denotes a signed variable-length encoded integer.

it would be nice to describe the encoding here as well, after all this
part of the docs is supposed to help understand how to parse such a
file.

> +
> +.. list-table::
> +   :widths: auto
> +
> +   * - ``MAGIC: [u8; 8]``
> +     - ``[145, 253, 96, 249, 196, 103, 88, 213]``.
> +   * - ``TABLE_LENGTH: u64*``
> +     - Total length of the contents inside the Directory.
> +   * - ``ENTRY_COUNT: u64*``
> +     - Count of entries present inside the Directory.
> +   * - ``ENTRIES``
> +     - Concatinated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout <entry-layout>` .) Repeated ``ENTRY_COUNT`` times.

concatenated

> +   * - ...
> +     - Further concatination of sequential directory blocks (Table Length, Entry Count and Entries).

concatenation

what makes directory blocks "sequential"?


> +   * - ``ROOT_START_OFFSET: u64``
> +     - Root marker (/) start offset. Stored as **little-endian**

the root marker is very important, as it points at the offset of the
directory where somebody parsing this file needs to start doing
recursive lookups..

so maybe this should actually be

MAGIC
[catalog data]
ROOT_START_OFFSET

where ROOT_START_OFFSET points at the start of the '/' catalog entry

and then we have

catalog entry:

> +   * - ``TABLE_LENGTH: u64*``
> +     - Total length of the catalog contents of this Directory.
> +   * - ``ENTRY_COUNT: u64*``
> +     - Count of directory entries
> +   * - ``ENTRIES``
> +     - Concatenated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout <entry-layout>` .), repeated ``ENTRY_COUNT`` times.

> +
> +.. _entry-layout:
> +
> +Except for files and directories, no additional payload is stored beyond the name and its length inside an ENTRY.
> +
> +.. list-table::
> +   :widths: auto
> +
> +   * - ``ENTRY_TYPE: u8``
> +     - The type of entry (see entry :ref:`types <entry-types>`).
> +   * - ``NAME_LENGTH: u64*``
> +     - Length of file or directory name. Stored as variable length encoded u64.
> +   * - ``NAME: [u8]``
> +     - The raw UTF-8 string bytes of the name.
> +   * - ``PAYLOAD``
> +     - Bytes based on ``ENTRY_TYPE``
> +
> +       * **Directory ('d')**: ``offset_back: u64*`` (Child-to-Parent offset distance)

the offset back probably requires more explanation

> +       * **File ('f')**: ``size: u64*`` followed by ``mtime: i64*``
> +.. _entry-types:
> +
> +Entry Types:
> +
> +.. list-table::
> +   :widths: auto
> +
> +   * - ``d``
> +     - Directory
> +   * - ``f``
> +     - File
> +   * - ``l``
> +     - Symlink
> +   * - ``h``
> +     - Hardlink
> +   * - ``b``
> +     - Block device
> +   * - ``c``
> +     - Char device
> +   * - ``p``
> +     - FIFO
> +   * - ``s``
> +     - Socket
>  .. _ppxar-format:
>  
>  Proxmox File Archive Format - Payload (``.ppxar``)
> -- 
> 2.47.3
> 
> 
> 
> 
> 
> 

Re: [PATCH proxmox-backup] fix #5748: docs: add `catalog.pcat1` format specification

[Shan Shaji]

On Tue Jul 14, 2026 at 11:38 AM CEST, Fabian Grünbichler wrote:
> On July 2, 2026 1:51 pm, Shan Shaji wrote:
>> Earlier proxmox catalog file format specification was not present
>> inside the docs. fixed it by adding format specification under file
>> formats.
>> 
>> Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=5748
>> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
>> ---
>>  changes since RFC: Thanks @Christian Ebner
>>  - Changed from graph to table layout.
>>  - Explicity mention about the variable length types. 
>>  - Denoted the root start offset as u64.
>>  - Add text description for each field.
>>  
>>  docs/file-formats.rst | 65 +++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 65 insertions(+)
>> 
>> diff --git a/docs/file-formats.rst b/docs/file-formats.rst
>> index ed3250d11..52cb20ee0 100644
>> --- a/docs/file-formats.rst
>> +++ b/docs/file-formats.rst
>> @@ -23,6 +23,71 @@ overhead introduced by the file payloads.
>>  
>>  .. graphviz:: meta-format-overview.dot
>>  
>> +.. _pcat1-format:
>> +
>> +Proxmox Catalog File Format (``.pcat1``)
>> +----------------------------------------
>
> this should go between pxar and mpxar, or below ppxar. there's arguments
> for both, but between mpxar and ppxar which always form a pair and never
> use catalog files is the wrong place ;)

makes sense, I will update it. 

>> +
>> +The asterisk notation indicates variable-length encoding: u64* denotes an unsigned variable-length encoded integer, while i64*
>> +denotes a signed variable-length encoded integer.
>
> it would be nice to describe the encoding here as well, after all this
> part of the docs is supposed to help understand how to parse such a
> file.

I will add that, thank you! Do you think a description explaining how we encode
it would be enough?

>> +
>> +.. list-table::
>> +   :widths: auto
>> +
>> +   * - ``MAGIC: [u8; 8]``
>> +     - ``[145, 253, 96, 249, 196, 103, 88, 213]``.
>> +   * - ``TABLE_LENGTH: u64*``
>> +     - Total length of the contents inside the Directory.
>> +   * - ``ENTRY_COUNT: u64*``
>> +     - Count of entries present inside the Directory.
>> +   * - ``ENTRIES``
>> +     - Concatinated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout <entry-layout>` .) Repeated ``ENTRY_COUNT`` times.
>
> concatenated

Thank you, I will update it.

>> +   * - ...
>> +     - Further concatination of sequential directory blocks (Table Length, Entry Count and Entries).
>
> concatenation
>
> what makes directory blocks "sequential"?

What I meant was like, the table length, number of entries and the data bytes are
written sequentially. Now that I think about it, I think it is wrong. The
reader could misinterpret it and think that the file can be read sequentially.

Thank you!

<MAGIC:8>|foo2_info|dir3_info|dir2_info|dir1_info|/
>
>> +   * - ``ROOT_START_OFFSET: u64``
>> +     - Root marker (/) start offset. Stored as **little-endian**
>
> the root marker is very important, as it points at the offset of the
> directory where somebody parsing this file needs to start doing
> recursive lookups..
>
> so maybe this should actually be
>
> MAGIC
> [catalog data]
> ROOT_START_OFFSET
>
> where ROOT_START_OFFSET points at the start of the '/' catalog entry
>
> and then we have
>
> catalog entry:

Did you meant catalog data here? In the above suggestion you mentioned
catalog data between the MAGIC and the root start offset.  

>> +   * - ``TABLE_LENGTH: u64*``
>> +     - Total length of the catalog contents of this Directory.
>> +   * - ``ENTRY_COUNT: u64*``
>> +     - Count of directory entries
>> +   * - ``ENTRIES``
>> +     - Concatenated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout <entry-layout>` .), repeated ``ENTRY_COUNT`` times.
>
>> +
>> +.. _entry-layout:
>> +
>> +Except for files and directories, no additional payload is stored beyond the name and its length inside an ENTRY.
>> +
>> +.. list-table::
>> +   :widths: auto
>> +
>> +   * - ``ENTRY_TYPE: u8``
>> +     - The type of entry (see entry :ref:`types <entry-types>`).
>> +   * - ``NAME_LENGTH: u64*``
>> +     - Length of file or directory name. Stored as variable length encoded u64.
>> +   * - ``NAME: [u8]``
>> +     - The raw UTF-8 string bytes of the name.
>> +   * - ``PAYLOAD``
>> +     - Bytes based on ``ENTRY_TYPE``
>> +
>> +       * **Directory ('d')**: ``offset_back: u64*`` (Child-to-Parent offset distance)
>
> the offset back probably requires more explanation

I will update it. Thank you!

>> +       * **File ('f')**: ``size: u64*`` followed by ``mtime: i64*``
>> +.. _entry-types:
>> +
>> +Entry Types:
>> +
>> +.. list-table::
>> +   :widths: auto
>> +
>> +   * - ``d``
>> +     - Directory
>> +   * - ``f``
>> +     - File
>> +   * - ``l``
>> +     - Symlink
>> +   * - ``h``
>> +     - Hardlink
>> +   * - ``b``
>> +     - Block device
>> +   * - ``c``
>> +     - Char device
>> +   * - ``p``
>> +     - FIFO
>> +   * - ``s``
>> +     - Socket
>>  .. _ppxar-format:
>>  
>>  Proxmox File Archive Format - Payload (``.ppxar``)
>> -- 
>> 2.47.3
>> 
>> 
>> 
>> 
>> 
>>