From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 69EC91FF0E4 for ; Tue, 14 Jul 2026 11:38:57 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4D65F21354; Tue, 14 Jul 2026 11:38:56 +0200 (CEST) Date: Tue, 14 Jul 2026 11:38:47 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [PATCH proxmox-backup] fix #5748: docs: add `catalog.pcat1` format specification To: pbs-devel@lists.proxmox.com, Shan Shaji References: <20260702115115.195035-1-s.shaji@proxmox.com> In-Reply-To: <20260702115115.195035-1-s.shaji@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1784019611.wtdtzouh6v.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1784021914836 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.259 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: PBUINKXRCP7G4PS3XCXTDMM3ZR7PYZD6 X-Message-ID-Hash: PBUINKXRCP7G4PS3XCXTDMM3ZR7PYZD6 X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On July 2, 2026 1:51 pm, Shan Shaji wrote: > Earlier proxmox catalog file format specification was not present > inside the docs. fixed it by adding format specification under file > formats. >=20 > Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=3D5748 > Signed-off-by: Shan Shaji > --- > changes since RFC: Thanks @Christian Ebner > - Changed from graph to table layout. > - Explicity mention about the variable length types.=20 > - Denoted the root start offset as u64. > - Add text description for each field. > =20 > docs/file-formats.rst | 65 +++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 65 insertions(+) >=20 > diff --git a/docs/file-formats.rst b/docs/file-formats.rst > index ed3250d11..52cb20ee0 100644 > --- a/docs/file-formats.rst > +++ b/docs/file-formats.rst > @@ -23,6 +23,71 @@ overhead introduced by the file payloads. > =20 > .. graphviz:: meta-format-overview.dot > =20 > +.. _pcat1-format: > + > +Proxmox Catalog File Format (``.pcat1``) > +---------------------------------------- this should go between pxar and mpxar, or below ppxar. there's arguments for both, but between mpxar and ppxar which always form a pair and never use catalog files is the wrong place ;) > + > +The asterisk notation indicates variable-length encoding: u64* denotes a= n unsigned variable-length encoded integer, while i64* > +denotes a signed variable-length encoded integer. it would be nice to describe the encoding here as well, after all this part of the docs is supposed to help understand how to parse such a file. > + > +.. list-table:: > + :widths: auto > + > + * - ``MAGIC: [u8; 8]`` > + - ``[145, 253, 96, 249, 196, 103, 88, 213]``. > + * - ``TABLE_LENGTH: u64*`` > + - Total length of the contents inside the Directory. > + * - ``ENTRY_COUNT: u64*`` > + - Count of entries present inside the Directory. > + * - ``ENTRIES`` > + - Concatinated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout= ` .) Repeated ``ENTRY_COUNT`` times. concatenated > + * - ... > + - Further concatination of sequential directory blocks (Table Lengt= h, Entry Count and Entries). concatenation what makes directory blocks "sequential"? > + * - ``ROOT_START_OFFSET: u64`` > + - Root marker (/) start offset. Stored as **little-endian** the root marker is very important, as it points at the offset of the directory where somebody parsing this file needs to start doing recursive lookups.. so maybe this should actually be MAGIC [catalog data] ROOT_START_OFFSET where ROOT_START_OFFSET points at the start of the '/' catalog entry and then we have catalog entry: > + * - ``TABLE_LENGTH: u64*`` > + - Total length of the catalog contents of this Directory. > + * - ``ENTRY_COUNT: u64*`` > + - Count of directory entries > + * - ``ENTRIES`` > + - Concatenated ``ENTRY`` structures (see the ``ENTRY`` :ref:`layout= ` .), repeated ``ENTRY_COUNT`` times. > + > +.. _entry-layout: > + > +Except for files and directories, no additional payload is stored beyond= the name and its length inside an ENTRY. > + > +.. list-table:: > + :widths: auto > + > + * - ``ENTRY_TYPE: u8`` > + - The type of entry (see entry :ref:`types `). > + * - ``NAME_LENGTH: u64*`` > + - Length of file or directory name. Stored as variable length encod= ed u64. > + * - ``NAME: [u8]`` > + - The raw UTF-8 string bytes of the name. > + * - ``PAYLOAD`` > + - Bytes based on ``ENTRY_TYPE`` > + > + * **Directory ('d')**: ``offset_back: u64*`` (Child-to-Parent off= set distance) the offset back probably requires more explanation > + * **File ('f')**: ``size: u64*`` followed by ``mtime: i64*`` > +.. _entry-types: > + > +Entry Types: > + > +.. list-table:: > + :widths: auto > + > + * - ``d`` > + - Directory > + * - ``f`` > + - File > + * - ``l`` > + - Symlink > + * - ``h`` > + - Hardlink > + * - ``b`` > + - Block device > + * - ``c`` > + - Char device > + * - ``p`` > + - FIFO > + * - ``s`` > + - Socket > .. _ppxar-format: > =20 > Proxmox File Archive Format - Payload (``.ppxar``) > --=20 > 2.47.3 >=20 >=20 >=20 >=20 >=20 >=20