Re: [PATCH proxmox-backup] fix #5748: docs: add `catalog.pcat1` format specification

["Shan Shaji" <s.shaji@proxmox.com>]

On Tue Jul 14, 2026 at 11:38 AM CEST, Fabian Grünbichler wrote:
> On July 2, 2026 1:51 pm, Shan Shaji wrote:
>> Earlier proxmox catalog file format specification was not present
>> inside the docs. fixed it by adding format specification under file
>> formats.
>> 
>> Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=5748
>> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
>> ---
>>  changes since RFC: Thanks @Christian Ebner
>>  - Changed from graph to table layout.
>>  - Explicity mention about the variable length types. 
>>  - Denoted the root start offset as u64.
>>  - Add text description for each field.
>>  
>>  docs/file-formats.rst | 65 +++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 65 insertions(+)
>> 
>> diff --git a/docs/file-formats.rst b/docs/file-formats.rst
>> index ed3250d11..52cb20ee0 100644
>> --- a/docs/file-formats.rst
>> +++ b/docs/file-formats.rst
>> @@ -23,6 +23,71 @@ overhead introduced by the file payloads.
>>  
>>  .. graphviz:: meta-format-overview.dot
>>  
>> +.. _pcat1-format:
>> +
>> +Proxmox Catalog File Format (``.pcat1``)
>> +----------------------------------------
>
> this should go between pxar and mpxar, or below ppxar. there's arguments
> for both, but between mpxar and ppxar which always form a pair and never
> use catalog files is the wrong place ;)

makes sense, I will update it. 

>> +
>> +The asterisk notation indicates variable-length encoding: u64* denotes an unsigned variable-length encoded integer, while i64*
>> +denotes a signed variable-length encoded integer.
>
> it would be nice to describe the encoding here as well, after all this
> part of the docs is supposed to help understand how to parse such a
> file.

I will add that, thank you! Do you think a description explaining how we encode
it would be enough?

>> +
>> +.. list-table::
>> +   :widths: auto
>> +
>> +   * - ``MAGIC: [u8; 8]``
>> +     - ``[145, 253, 96, 249, 196, 103, 88, 213]``.
>> +   * - ``TABLE_LENGTH: u64*``
>> +     - Total length of the contents inside the Directory.
>> +   * - ``ENTRY_COUNT: u64*``
>> +     - Count of entries present inside the Directory.
>> +   * - ``ENTRIES``
>> +     - Concatinated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout <entry-layout>` .) Repeated ``ENTRY_COUNT`` times.
>
> concatenated

Thank you, I will update it.

>> +   * - ...
>> +     - Further concatination of sequential directory blocks (Table Length, Entry Count and Entries).
>
> concatenation
>
> what makes directory blocks "sequential"?

What I meant was like, the table length, number of entries and the data bytes are
written sequentially. Now that I think about it, I think it is wrong. The
reader could misinterpret it and think that the file can be read sequentially.

Thank you!

<MAGIC:8>|foo2_info|dir3_info|dir2_info|dir1_info|/
>
>> +   * - ``ROOT_START_OFFSET: u64``
>> +     - Root marker (/) start offset. Stored as **little-endian**
>
> the root marker is very important, as it points at the offset of the
> directory where somebody parsing this file needs to start doing
> recursive lookups..
>
> so maybe this should actually be
>
> MAGIC
> [catalog data]
> ROOT_START_OFFSET
>
> where ROOT_START_OFFSET points at the start of the '/' catalog entry
>
> and then we have
>
> catalog entry:

Did you meant catalog data here? In the above suggestion you mentioned
catalog data between the MAGIC and the root start offset.  

>> +   * - ``TABLE_LENGTH: u64*``
>> +     - Total length of the catalog contents of this Directory.
>> +   * - ``ENTRY_COUNT: u64*``
>> +     - Count of directory entries
>> +   * - ``ENTRIES``
>> +     - Concatenated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout <entry-layout>` .), repeated ``ENTRY_COUNT`` times.
>
>> +
>> +.. _entry-layout:
>> +
>> +Except for files and directories, no additional payload is stored beyond the name and its length inside an ENTRY.
>> +
>> +.. list-table::
>> +   :widths: auto
>> +
>> +   * - ``ENTRY_TYPE: u8``
>> +     - The type of entry (see entry :ref:`types <entry-types>`).
>> +   * - ``NAME_LENGTH: u64*``
>> +     - Length of file or directory name. Stored as variable length encoded u64.
>> +   * - ``NAME: [u8]``
>> +     - The raw UTF-8 string bytes of the name.
>> +   * - ``PAYLOAD``
>> +     - Bytes based on ``ENTRY_TYPE``
>> +
>> +       * **Directory ('d')**: ``offset_back: u64*`` (Child-to-Parent offset distance)
>
> the offset back probably requires more explanation

I will update it. Thank you!

>> +       * **File ('f')**: ``size: u64*`` followed by ``mtime: i64*``
>> +.. _entry-types:
>> +
>> +Entry Types:
>> +
>> +.. list-table::
>> +   :widths: auto
>> +
>> +   * - ``d``
>> +     - Directory
>> +   * - ``f``
>> +     - File
>> +   * - ``l``
>> +     - Symlink
>> +   * - ``h``
>> +     - Hardlink
>> +   * - ``b``
>> +     - Block device
>> +   * - ``c``
>> +     - Char device
>> +   * - ``p``
>> +     - FIFO
>> +   * - ``s``
>> +     - Socket
>>  .. _ppxar-format:
>>  
>>  Proxmox File Archive Format - Payload (``.ppxar``)
>> -- 
>> 2.47.3
>> 
>> 
>> 
>> 
>> 
>>