Re: [RFC] Native macOS PBS client: filesystem metadata and tested prototype

["Adam Okoń" <adamokon@gmail.com>]

Hi Chris, hi list,

Following the 30 October native-PXAR prototype update, I have
incorporated a small set of recent pbs-devel client fixes into both
maintained macOS branches.

The native format proposal remains here:
https://github.com/adamokon/proxmox-backup-macos/pull/2
The existing Homebrew branch is tracked by PR #1:
https://github.com/adamokon/proxmox-backup-macos/pull/1

Changes and attribution:
- Key handling now validates plaintext key lengths and uses checked
conversion after decryption instead of panicking. This follows
upstream a23876b82f2c and 6cd68788c200. IV and authentication-tag
lengths are separate constants, and truncated encrypted framing is
rejected before requesting a password, following the 10 November
discussion.
- PXAR reusable-chunk lookup now stops at an equal exclusive range
end, following accepted upstream 3a7518533b2a, with boundary
regression cases.
- Datastore status omits percentages when total space is zero and
returns errors for malformed counters, following your 6 November
status-rendering patch. Percentage arithmetic additionally uses u128
intermediates to avoid overflow for valid u64 counters.
- The declared minimum Rust version is 1.91, with an explicit CI
compilation check.

Upstream references and the precise scope are recorded in:
https://github.com/adamokon/proxmox-backup-macos/blob/codex/native-pxar-metadata/docs/upstream-backports-20261111.md

Validation on 11 November:
- Isolated Apple Silicon runs passed 27 tests on the native-PXAR
branch and 13 on a clean, patched checkout of the Homebrew branch. The
latter also passed an optimized native build.
- Actual CLI tests reject synthetic 0-, 31- and 33-byte keys with an
ordinary error; a valid 32-byte control succeeds.
- Fresh encrypted backup/restore against PBS passed on both branches.
Comparisons covered file bytes, hardlinks, symlinks, modes,
mtime/birthtime, BSD flags, ordered ACLs, binary xattrs, FinderInfo
and a resource fork larger than 1 MiB. All ten final test snapshots
were independently confirmed on PBS with verification state "ok".
- Both Linux CI runs passed, including regression tests, client
compilation and an explicit check with Rust 1.91:
  Homebrew branch:
https://github.com/adamokon/proxmox-backup-macos/actions/runs/43049710818
  Native PXAR branch:
https://github.com/adamokon/proxmox-backup-macos/actions/runs/43049711367

This is a scoped maintenance backport onto the v4.2.8-based prototype,
not a complete rebase to v4.2.10. The experimental PXAR wire layout is
unchanged by this update. The format identifiers remain unallocated,
Windows filesystem APIs and Intel macOS remain untested, and
independent human review is still required. The implementation and
this update used extensive AI assistance.

Thanks to the upstream authors for the fixes.

Best regards,
Adam Okoń

On Fri, 30 Oct 2026 16:14:32 -0500, Adam Okoń <adamokon@gmail.com> wrote:
> Hi Chris,
>
> Thank you for the architectural guidance. I have implemented an experimental revision following the native PXAR approach you suggested.
>
> The changes relative to the initial macOS prototype are available here:
> https://github.com/adamokon/proxmox-backup-macos/pull/2
>
> The wire layout and review boundaries are documented here:
> https://github.com/adamokon/proxmox-backup-macos/blob/codex/native-pxar-metadata/docs/native-pxar-draft.md
>
> The revised implementation adds native metadata, attribute descriptor and payload records to the PXAR encoder/decoder. It supports both unified .pxar archives and split .mpxar/.ppxar archives. In the split representation, descriptors, lengths and content digests remain in .mpxar, while attribute values and resource forks are streamed into .ppxar. Metadata-only traversal does not require the payload stream, and selective restoration can fetch the native payloads for the selected entries.
>
> New backups no longer use the synthetic xattr envelope, auxiliary archive or metadata map from the initial prototype. The new reader retains support for that prototype's backups, and the previous PXAR v1/v2 encoding and decoding paths remain available.
>
> For Darwin, the records cover birthtime, BSD flags, ordered UUID ACL entries and native xattrs/resource forks. The draft also represents Windows file attributes, FILETIMEs, security descriptors, reparse data and named streams. Windows coverage is currently synthetic wire-format testing; Windows filesystem capture/restore APIs have not been implemented or tested.
>
> Validation completed for this revision:
> - 16 isolated Apple Silicon tests passed, including actual unified/split filesystem restoration, selective subdirectory restoration, metadata-only and random access, malformed/truncated/corrupt inputs, and the original upstream old-timestamp compatibility fixture.
> - Linux CI passed: PBS format regressions, PXAR library and native wire tests, path matching, endian/mmap checks and full Linux client compilation:
>   https://github.com/adamokon/proxmox-backup-macos/actions/runs/41128035455
> - Encrypted backup and restore against a real PBS server passed for the legacy prototype and the new unified/split representations. Comparisons covered file bytes, hardlinks, symlink targets, permissions, timestamps, BSD flags, ACLs, binary xattrs, FinderInfo and a resource fork larger than 1 MiB. All five final test snapshots were independently confirmed on PBS with verification state "ok".
> - In the split fixture, .mpxar was 1779 bytes for approximately 5 MiB of payloads. Repeated unified/split backups reused 100% of archive data chunks.
>
> The draft uses deliberately unallocated experimental version/record identifiers, pending upstream agreement. One performance limitation remains: files carrying native attributes are conservatively re-read/re-encoded rather than eliding reads through split payload-range reuse. Chunk deduplication remains effective, but that reuse optimization needs further work.
>
> The implementation continues to use extensive AI assistance and still requires independent human review. Intel macOS has not been tested. Thank you also for clarifying the current endian_trait usage and the planned dependency removal; the inherited compatibility fork should be considered separately from the native format proposal.
>
> Could you comment on whether this revised framing and payload placement fit the intended PXAR architecture? In particular, are the proposed platform records and Windows metadata categories an appropriate starting point, and what would you change before preparing a cleaned-up, cross-repository patch series for the mailing list?
>
> Best regards,
> Adam Okoń
>
>
> On Thu, 29 Oct 2026 11:05:46 +0100, Christian Ebner <c.ebner@proxmox.com> wrote:
> > Hi,
> >
> > On 10/27/26 10:31 AM, Adam Okoń wrote:
> > > Hello,
> > >
> > > I would appreciate architectural feedback on a native macOS port of
> > > proxmox-backup-client, based on PBS v4.2.8. The prototype runs without
> > > Docker or a Linux guest and keeps the usual backup/restore CLI, PBS
> > > chunking, deduplication and client-side encryption.
> > >
> > > Source and review workspace:
> > > https://github.com/adamokon/proxmox-backup-macos/pull/1
> >
> > thanks for reaching out. As an upfront disclaimer: please note that we
> > can only accept high quality code contributions which keep the codebase
> > stable and maintainable.
> > > The main design question is preservation of Darwin filesystem
> > > metadata. The prototype uses a bounded, versioned envelope in standard
> > > PXAR xattr records to describe ordered UUID ACLs, birthtime, BSD flags
> > > and native attribute content references. Attribute bytes, including
> > > large resource forks, are content-addressed regular files in an
> > > additional standard PXAR archive in the same snapshot. A
> > > manifest-listed map associates it with the primary archives. The
> > > server protocol and PXAR record format are unchanged. Temporary
> > > spooling is authenticated-encrypted with an ephemeral local key, then
> > > the ordinary PBS encryption and deduplication pipeline processes the
> > > upload stream.
> >
> > While not being an expert when it comes to macOS and it's file
> > attributes, I don't think the chosen initial implementation approach is
> > the best fitting one.
> >
> > Rather than trying to bypass changes to the pxar archive format, it
> > should rather be extended when needed, implementing native
> > encoder/decoder support for macOS specific attributes and additional
> > payloads. This should then cover both, regular .pxar archives as well as
> > split .mpxar/ppxar archives, while for the latter, the size of .mpxar is
> > ideally kept small, placing larger payloads into the .ppxar file.
> > Ideally the .mpxar only contains metadata needed for file change
> > detection between subsequent backup runs and the data needed for fast
> > content introspection, only fetching .ppxar content chunks when
> > required, keeping metadata lookups fast.
> >
> > Since this will require a new pxar format version, ideally it would be
> > extended in a manner which covers windows filesystem attributes as well,
> > so no further format changes are required for that. Backwards
> > compatibility must be maintained, the newer implementation fully capable
> > of encoding and decoding the previous format versions.
> > > Native archive/restore tests cover Unicode filenames, symlinks,
> > > hardlinks, binary xattrs, FinderInfo, a resource fork larger than 1
> > > MiB, nanosecond times and destination name alias rejection before
> > > truncation. A fresh Homebrew source installation on Apple Silicon also
> > > passed backup and restore against a dedicated PBS test namespace, both
> > > unencrypted and client-side encrypted. Repeated backups reused 100% of
> > > primary and auxiliary archive data. Linux client compilation and
> > > format regression checks accompany the port.
> > >
> > > The work also separates client formats from Linux server dependencies
> > > and fixes native fd/xattr/time/UUID APIs and mmap alignment on 16 KiB
> > > pages. The inherited endian_trait ownership issue (RUSTSEC-2021-0039)
> > > is addressed by a documented local MIT-licensed compatibility fork
> > > using panic-safe array mapping and Copy-bound slices; it is not
> > > presented as a published upstream fix.
> >
> > Note that while PBS depends on endian_trait, there is currently no
> > codepath which does implement the Endian trait directly and would lead
> > to a panic. The dependency is however already been flagged internally as
> > to be phased out.
> >
> > > Current scope and limits: Apple Silicon is tested; Intel is not yet
> > > tested. FUSE mount/map remain Linux-only. There is no automatic iCloud
> > > placeholder download or atomic APFS snapshot. Protected system-managed
> > > xattrs can produce explicit restore warnings. This is an experimental
> > > prototype, not a claim of official support or a merge-ready packaging
> > > series.
> > >
> > > The implementation was developed with extensive AI assistance. Its
> > > provenance, source patches and actual test results are available for
> > > review. Proxmox confirmed on 26 October 2026 that my Individual CLA is
> > > on record for future contributions. No independent human code review
> > > is claimed at this stage.
> > >
> > > Before preparing a cleaned-up cross-repository patch series, would
> > > this native client scope and metadata representation fit the upstream
> > > architecture? I would welcome guidance on a preferred representation,
> > > required review and how to split the
> > > proxmox-backup/proxmox/pxar/pathpatterns changes.
> >
> > Patches have to be send to the developers mailing list for review
> > following the guidelines as outlined in
> > https://pbs.proxmox.com/wiki/Developer_Documentation. The patch series
> > can span multiple repositories, the patches subject prefixed according
> > to the repositories they belong to.
> >
> > Best regards,
> > Chris