Re: [RFC] Native macOS PBS client: filesystem metadata and tested prototype

["Adam Okoń" <adamokon@gmail.com>]

Hi Chris,

Thank you for the architectural guidance. I have implemented an
experimental revision following the native PXAR approach you
suggested.

The changes relative to the initial macOS prototype are available here:
https://github.com/adamokon/proxmox-backup-macos/pull/2

The wire layout and review boundaries are documented here:
https://github.com/adamokon/proxmox-backup-macos/blob/codex/native-pxar-metadata/docs/native-pxar-draft.md

The revised implementation adds native metadata, attribute descriptor
and payload records to the PXAR encoder/decoder. It supports both
unified .pxar archives and split .mpxar/.ppxar archives. In the split
representation, descriptors, lengths and content digests remain in
.mpxar, while attribute values and resource forks are streamed into
.ppxar. Metadata-only traversal does not require the payload stream,
and selective restoration can fetch the native payloads for the
selected entries.

New backups no longer use the synthetic xattr envelope, auxiliary
archive or metadata map from the initial prototype. The new reader
retains support for that prototype's backups, and the previous PXAR
v1/v2 encoding and decoding paths remain available.

For Darwin, the records cover birthtime, BSD flags, ordered UUID ACL
entries and native xattrs/resource forks. The draft also represents
Windows file attributes, FILETIMEs, security descriptors, reparse data
and named streams. Windows coverage is currently synthetic wire-format
testing; Windows filesystem capture/restore APIs have not been
implemented or tested.

Validation completed for this revision:
- 16 isolated Apple Silicon tests passed, including actual
unified/split filesystem restoration, selective subdirectory
restoration, metadata-only and random access,
malformed/truncated/corrupt inputs, and the original upstream
old-timestamp compatibility fixture.
- Linux CI passed: PBS format regressions, PXAR library and native
wire tests, path matching, endian/mmap checks and full Linux client
compilation:
  https://github.com/adamokon/proxmox-backup-macos/actions/runs/41128035455
- Encrypted backup and restore against a real PBS server passed for
the legacy prototype and the new unified/split representations.
Comparisons covered file bytes, hardlinks, symlink targets,
permissions, timestamps, BSD flags, ACLs, binary xattrs, FinderInfo
and a resource fork larger than 1 MiB. All five final test snapshots
were independently confirmed on PBS with verification state "ok".
- In the split fixture, .mpxar was 1779 bytes for approximately 5 MiB
of payloads. Repeated unified/split backups reused 100% of archive
data chunks.

The draft uses deliberately unallocated experimental version/record
identifiers, pending upstream agreement. One performance limitation
remains: files carrying native attributes are conservatively
re-read/re-encoded rather than eliding reads through split
payload-range reuse. Chunk deduplication remains effective, but that
reuse optimization needs further work.

The implementation continues to use extensive AI assistance and still
requires independent human review. Intel macOS has not been tested.
Thank you also for clarifying the current endian_trait usage and the
planned dependency removal; the inherited compatibility fork should be
considered separately from the native format proposal.

Could you comment on whether this revised framing and payload
placement fit the intended PXAR architecture? In particular, are the
proposed platform records and Windows metadata categories an
appropriate starting point, and what would you change before preparing
a cleaned-up, cross-repository patch series for the mailing list?

Best regards,
Adam Okoń


On Thu, 29 Oct 2026 11:05:46 +0100, Christian Ebner <c.ebner@proxmox.com> wrote:
> Hi,
>
> On 10/27/26 10:31 AM, Adam Okoń wrote:
> > Hello,
> >
> > I would appreciate architectural feedback on a native macOS port of
> > proxmox-backup-client, based on PBS v4.2.8. The prototype runs without
> > Docker or a Linux guest and keeps the usual backup/restore CLI, PBS
> > chunking, deduplication and client-side encryption.
> >
> > Source and review workspace:
> > https://github.com/adamokon/proxmox-backup-macos/pull/1
>
> thanks for reaching out. As an upfront disclaimer: please note that we
> can only accept high quality code contributions which keep the codebase
> stable and maintainable.
> > The main design question is preservation of Darwin filesystem
> > metadata. The prototype uses a bounded, versioned envelope in standard
> > PXAR xattr records to describe ordered UUID ACLs, birthtime, BSD flags
> > and native attribute content references. Attribute bytes, including
> > large resource forks, are content-addressed regular files in an
> > additional standard PXAR archive in the same snapshot. A
> > manifest-listed map associates it with the primary archives. The
> > server protocol and PXAR record format are unchanged. Temporary
> > spooling is authenticated-encrypted with an ephemeral local key, then
> > the ordinary PBS encryption and deduplication pipeline processes the
> > upload stream.
>
> While not being an expert when it comes to macOS and it's file
> attributes, I don't think the chosen initial implementation approach is
> the best fitting one.
>
> Rather than trying to bypass changes to the pxar archive format, it
> should rather be extended when needed, implementing native
> encoder/decoder support for macOS specific attributes and additional
> payloads. This should then cover both, regular .pxar archives as well as
> split .mpxar/ppxar archives, while for the latter, the size of .mpxar is
> ideally kept small, placing larger payloads into the .ppxar file.
> Ideally the .mpxar only contains metadata needed for file change
> detection between subsequent backup runs and the data needed for fast
> content introspection, only fetching .ppxar content chunks when
> required, keeping metadata lookups fast.
>
> Since this will require a new pxar format version, ideally it would be
> extended in a manner which covers windows filesystem attributes as well,
> so no further format changes are required for that. Backwards
> compatibility must be maintained, the newer implementation fully capable
> of encoding and decoding the previous format versions.
> > Native archive/restore tests cover Unicode filenames, symlinks,
> > hardlinks, binary xattrs, FinderInfo, a resource fork larger than 1
> > MiB, nanosecond times and destination name alias rejection before
> > truncation. A fresh Homebrew source installation on Apple Silicon also
> > passed backup and restore against a dedicated PBS test namespace, both
> > unencrypted and client-side encrypted. Repeated backups reused 100% of
> > primary and auxiliary archive data. Linux client compilation and
> > format regression checks accompany the port.
> >
> > The work also separates client formats from Linux server dependencies
> > and fixes native fd/xattr/time/UUID APIs and mmap alignment on 16 KiB
> > pages. The inherited endian_trait ownership issue (RUSTSEC-2021-0039)
> > is addressed by a documented local MIT-licensed compatibility fork
> > using panic-safe array mapping and Copy-bound slices; it is not
> > presented as a published upstream fix.
>
> Note that while PBS depends on endian_trait, there is currently no
> codepath which does implement the Endian trait directly and would lead
> to a panic. The dependency is however already been flagged internally as
> to be phased out.
>
> > Current scope and limits: Apple Silicon is tested; Intel is not yet
> > tested. FUSE mount/map remain Linux-only. There is no automatic iCloud
> > placeholder download or atomic APFS snapshot. Protected system-managed
> > xattrs can produce explicit restore warnings. This is an experimental
> > prototype, not a claim of official support or a merge-ready packaging
> > series.
> >
> > The implementation was developed with extensive AI assistance. Its
> > provenance, source patches and actual test results are available for
> > review. Proxmox confirmed on 26 October 2026 that my Individual CLA is
> > on record for future contributions. No independent human code review
> > is claimed at this stage.
> >
> > Before preparing a cleaned-up cross-repository patch series, would
> > this native client scope and metadata representation fit the upstream
> > architecture? I would welcome guidance on a preferred representation,
> > required review and how to split the
> > proxmox-backup/proxmox/pxar/pathpatterns changes.
>
> Patches have to be send to the developers mailing list for review
> following the guidelines as outlined in
> https://pbs.proxmox.com/wiki/Developer_Documentation. The patch series
> can span multiple repositories, the patches subject prefixed according
> to the repositories they belong to.
>
> Best regards,
> Chris