From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C369E1FF0B3 for ; Wed, 11 Nov 2026 16:39:57 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 8A2F621615; Wed, 11 Nov 2026 16:39:57 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; t=1794411585; cv=none; d=google.com; s=arc-20260327; b=Z+yxVZB9oMkFQLWh0egctRudj9+TcgdEVU6SsPJTPlMiVuXZ6QgORI/p6zMXM9EPm0 WzogfQ7vFQCMBPuKHRClM6nOkWU/pVqUizONsnB6ejT3mq11QxHoL5HXq6YaVw/TGkyU WQgRfbk04omwXoRk0IeVQv29qZikQXfGYkM/6Gu+Pvv284rIAeUwd1vgXjmysjDOGd+E 59Vlh0CBiX0hxkLGvk53h5cNz1hOkTh+87FAYFg/OUWklvt9ukWkXujuR0DrBZKjlrXx Rtj1AnrIeWvEBn36qvqM4lFWqq6Bf6/gHx/I7LXxIYSg8g4Fe9vg9G/dMf1SYG148OEt w1Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:cc:to:subject:message-id:date:references :in-reply-to:from:mime-version:dkim-signature; bh=qQXTdU+toqJl87dLgFcIJ5sTuC9yha55DQzCs2b70EE=; fh=x735zFtK7d+S3igtQIHSkrR20uCSKtnfpmFHdvkFDH0=; b=ZEwty6SqVW5azJuhqCRl97WE+vzNyUXVNrwZ5vC8NF+v0KNwObXjmDIw+2NUkeab0J Qta0e4J04Xviy0PLbaBZ2SvKJaFuYmyNKJvRoTrZQXJA/E8/HWXIIZrIHsftsW6z3jR4 jSNi+2pKtfbPZ/rbogJh5v/MC4UMmkv4aZRGGvvV0ID/MTysia42zmV5+9UoEol49mqP t+x8prpa47Hlaswvbu9Gu9xUnwyBQubX6nAJRftv9R5mssPcprtykSBwYeQYs8HHVymn 41mVd68qJS/gZ91FlPcRqHWdGatfcinSz+qltqdLdhRsQ/rSFEMyTk1XxwyQLIhje5ky woRQ==; darn=lists.proxmox.com ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794411585; x=1795016385; darn=lists.proxmox.com; h=content-transfer-encoding:content-type:cc:to:subject:message-id :date:references:in-reply-to:from:mime-version:from:to:cc:subject :date:message-id:reply-to:content-type; bh=qQXTdU+toqJl87dLgFcIJ5sTuC9yha55DQzCs2b70EE=; b=ys60cayEcBcq0REUipvy9LU5VZx1fsmyzXLZBWYUi9gVTehI7O8ZAuGf1Tn8o+Vtvs mwUgP9BDZ7xg1hkQjQ1J+YltICHZ5ul5qDskhixpFj1pXY/He7I4vDH5yMiN9iFk9Vfj d7LPwflAEZlQ98S5RQGV5MgZb5AETucephrLtJ52osZ83+hKOK/iTo0wSd58sigbSWx2 S/JX3CdjNoZx462GjXBz8NMzKuvR8k3HVJRr6BTH9J5v7lRpduxjH+mC3oumKVrMWO7f tg0MyRxhtB24DLiVDxJJfoJftKeCWIXG94nQk1+mOjMNlxYLo3+et5U62U1A5Gy1HHHD 5Sqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794411585; x=1795016385; h=content-transfer-encoding:content-type:cc:to:subject:message-id :date:references:in-reply-to:from:mime-version:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=qQXTdU+toqJl87dLgFcIJ5sTuC9yha55DQzCs2b70EE=; b=fI/IbAClGqzryiDM6fkeKP7qhYTeoL3hX0UKiihZgItkwqf+E3hcWpl1BOnTTc7bmL n895jzfvFjYZViRyEBmwccLN4dM2rKkw/hmV7nuFy1xtQAmwCHR0D9VBwOwfMvYQj9LV UsAQByPIupDv5Xj0N42LPg6PjPDqyBuwbJCLP9U/522QfgiHWxWPlo7Jh1+9HSYSFhLp APjOF2DM4Korge7tejCub2xASDF9CNFEO5goNCpayzmKrPWaW3TXhO7YPH7qmKX526+A M3HiCu/rdspPmJ08tDScBl1zqch8O1oX01+IK8rmSqAearz2I27o6RjLWJ+rer0/0XGb Z01g== X-Gm-Message-State: APuU4Z8R9gf3BmRWbx4vU8uHIRLktZgaT6unsJ2RtZQveGsCYQjAVTfL LklL6tFm/UGQJW2UjC7g0uPFwZqlll5QSREsOtzhJ5tDfbGRcqd/CSdaepJzFCWQ0TyTUoNPAVt IXNoHby8wvcG98tRbcxESdEkP+fu5mwM= X-Gm-Gg: AYBFou0Uuw9uc6O8bHgV4FIWIwU0AD+PRrtJfGfNqpvTtSTJLVzP93uYMJrpE7/UYIi JyU1w1TOrPSEZ3r5qExFmxpIW6fq6q/G34UovqjwcxofjHsmbQaIP1KcGW6M3KoaOJA+Q20oSga eX5/Tp5sHXURG5BVBTJ6eyTYr97b2kVLDJQUw+5bsXx2Mbljn9ejLLtKRl6vDpRUYyN5NcWhPMA 1lhH/yqJKo6wvoPPwSUMGdud3bpVtg+P1fug7EuGYP4sq1p/irBRwQ5kEkboC+ynK1HnyRsolKp 1XbGnqdqM3GINcmdUskJ/E0yS1HL1gfC6Xw8UZsdRVC2a3xHCMjiaz/m+Z/9c+c7EId2wCBrHVZ lnEZKILj/k02EQJZwWy07ZVZaJUm8pe8CNupFjdV5/NYjyTjHQ3LMcfhmpeDrkBnGiMY= X-Received: by 2002:a17:90b:3c46:b0:3ac:7116:434a with SMTP id 98e67ed59e1d1-3b182efb2ecmr2705994a91.15.1794411584810; Wed, 11 Nov 2026 07:39:44 -0800 (PST) MIME-Version: 1.0 From: =?UTF-8?Q?Adam_Oko=C5=84?= In-Reply-To: References: Date: Wed, 11 Nov 2026 09:39:43 -0600 X-Gm-Features: AclHuK9yHyiJ_h245jInfWEKptm_VQ5bxsDs2Wg91Ead0lu48lLU-nw0DFcwg3k Message-ID: Subject: Re: [RFC] Native macOS PBS client: filesystem metadata and tested prototype To: Christian Ebner Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RM3U5EA3OMCEQMRRHHAJ4AA7FTXIY2FA X-Message-ID-Hash: RM3U5EA3OMCEQMRRHHAJ4AA7FTXIY2FA X-MailFrom: adamokon@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pbs-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Chris, hi list, Following the 30 October native-PXAR prototype update, I have incorporated a small set of recent pbs-devel client fixes into both maintained macOS branches. The native format proposal remains here: https://github.com/adamokon/proxmox-backup-macos/pull/2 The existing Homebrew branch is tracked by PR #1: https://github.com/adamokon/proxmox-backup-macos/pull/1 Changes and attribution: - Key handling now validates plaintext key lengths and uses checked conversion after decryption instead of panicking. This follows upstream a23876b82f2c and 6cd68788c200. IV and authentication-tag lengths are separate constants, and truncated encrypted framing is rejected before requesting a password, following the 10 November discussion. - PXAR reusable-chunk lookup now stops at an equal exclusive range end, following accepted upstream 3a7518533b2a, with boundary regression cases. - Datastore status omits percentages when total space is zero and returns errors for malformed counters, following your 6 November status-rendering patch. Percentage arithmetic additionally uses u128 intermediates to avoid overflow for valid u64 counters. - The declared minimum Rust version is 1.91, with an explicit CI compilation check. Upstream references and the precise scope are recorded in: https://github.com/adamokon/proxmox-backup-macos/blob/codex/native-pxar-met= adata/docs/upstream-backports-20261111.md Validation on 11 November: - Isolated Apple Silicon runs passed 27 tests on the native-PXAR branch and 13 on a clean, patched checkout of the Homebrew branch. The latter also passed an optimized native build. - Actual CLI tests reject synthetic 0-, 31- and 33-byte keys with an ordinary error; a valid 32-byte control succeeds. - Fresh encrypted backup/restore against PBS passed on both branches. Comparisons covered file bytes, hardlinks, symlinks, modes, mtime/birthtime, BSD flags, ordered ACLs, binary xattrs, FinderInfo and a resource fork larger than 1 MiB. All ten final test snapshots were independently confirmed on PBS with verification state "ok". - Both Linux CI runs passed, including regression tests, client compilation and an explicit check with Rust 1.91: Homebrew branch: https://github.com/adamokon/proxmox-backup-macos/actions/runs/43049710818 Native PXAR branch: https://github.com/adamokon/proxmox-backup-macos/actions/runs/43049711367 This is a scoped maintenance backport onto the v4.2.8-based prototype, not a complete rebase to v4.2.10. The experimental PXAR wire layout is unchanged by this update. The format identifiers remain unallocated, Windows filesystem APIs and Intel macOS remain untested, and independent human review is still required. The implementation and this update used extensive AI assistance. Thanks to the upstream authors for the fixes. Best regards, Adam Oko=C5=84 On Fri, 30 Oct 2026 16:14:32 -0500, Adam Oko=C5=84 wro= te: > Hi Chris, > > Thank you for the architectural guidance. I have implemented an experimen= tal revision following the native PXAR approach you suggested. > > The changes relative to the initial macOS prototype are available here: > https://github.com/adamokon/proxmox-backup-macos/pull/2 > > The wire layout and review boundaries are documented here: > https://github.com/adamokon/proxmox-backup-macos/blob/codex/native-pxar-m= etadata/docs/native-pxar-draft.md > > The revised implementation adds native metadata, attribute descriptor and= payload records to the PXAR encoder/decoder. It supports both unified .pxa= r archives and split .mpxar/.ppxar archives. In the split representation, d= escriptors, lengths and content digests remain in .mpxar, while attribute v= alues and resource forks are streamed into .ppxar. Metadata-only traversal = does not require the payload stream, and selective restoration can fetch th= e native payloads for the selected entries. > > New backups no longer use the synthetic xattr envelope, auxiliary archive= or metadata map from the initial prototype. The new reader retains support= for that prototype's backups, and the previous PXAR v1/v2 encoding and dec= oding paths remain available. > > For Darwin, the records cover birthtime, BSD flags, ordered UUID ACL entr= ies and native xattrs/resource forks. The draft also represents Windows fil= e attributes, FILETIMEs, security descriptors, reparse data and named strea= ms. Windows coverage is currently synthetic wire-format testing; Windows fi= lesystem capture/restore APIs have not been implemented or tested. > > Validation completed for this revision: > - 16 isolated Apple Silicon tests passed, including actual unified/split = filesystem restoration, selective subdirectory restoration, metadata-only a= nd random access, malformed/truncated/corrupt inputs, and the original upst= ream old-timestamp compatibility fixture. > - Linux CI passed: PBS format regressions, PXAR library and native wire t= ests, path matching, endian/mmap checks and full Linux client compilation: > https://github.com/adamokon/proxmox-backup-macos/actions/runs/411280354= 55 > - Encrypted backup and restore against a real PBS server passed for the l= egacy prototype and the new unified/split representations. Comparisons cove= red file bytes, hardlinks, symlink targets, permissions, timestamps, BSD fl= ags, ACLs, binary xattrs, FinderInfo and a resource fork larger than 1 MiB.= All five final test snapshots were independently confirmed on PBS with ver= ification state "ok". > - In the split fixture, .mpxar was 1779 bytes for approximately 5 MiB of = payloads. Repeated unified/split backups reused 100% of archive data chunks= . > > The draft uses deliberately unallocated experimental version/record ident= ifiers, pending upstream agreement. One performance limitation remains: fil= es carrying native attributes are conservatively re-read/re-encoded rather = than eliding reads through split payload-range reuse. Chunk deduplication r= emains effective, but that reuse optimization needs further work. > > The implementation continues to use extensive AI assistance and still req= uires independent human review. Intel macOS has not been tested. Thank you = also for clarifying the current endian_trait usage and the planned dependen= cy removal; the inherited compatibility fork should be considered separatel= y from the native format proposal. > > Could you comment on whether this revised framing and payload placement f= it the intended PXAR architecture? In particular, are the proposed platform= records and Windows metadata categories an appropriate starting point, and= what would you change before preparing a cleaned-up, cross-repository patc= h series for the mailing list? > > Best regards, > Adam Oko=C5=84 > > > On Thu, 29 Oct 2026 11:05:46 +0100, Christian Ebner = wrote: > > Hi, > > > > On 10/27/26 10:31 AM, Adam Oko=C5=84 wrote: > > > Hello, > > > > > > I would appreciate architectural feedback on a native macOS port of > > > proxmox-backup-client, based on PBS v4.2.8. The prototype runs withou= t > > > Docker or a Linux guest and keeps the usual backup/restore CLI, PBS > > > chunking, deduplication and client-side encryption. > > > > > > Source and review workspace: > > > https://github.com/adamokon/proxmox-backup-macos/pull/1 > > > > thanks for reaching out. As an upfront disclaimer: please note that we > > can only accept high quality code contributions which keep the codebase > > stable and maintainable. > > > The main design question is preservation of Darwin filesystem > > > metadata. The prototype uses a bounded, versioned envelope in standar= d > > > PXAR xattr records to describe ordered UUID ACLs, birthtime, BSD flag= s > > > and native attribute content references. Attribute bytes, including > > > large resource forks, are content-addressed regular files in an > > > additional standard PXAR archive in the same snapshot. A > > > manifest-listed map associates it with the primary archives. The > > > server protocol and PXAR record format are unchanged. Temporary > > > spooling is authenticated-encrypted with an ephemeral local key, then > > > the ordinary PBS encryption and deduplication pipeline processes the > > > upload stream. > > > > While not being an expert when it comes to macOS and it's file > > attributes, I don't think the chosen initial implementation approach is > > the best fitting one. > > > > Rather than trying to bypass changes to the pxar archive format, it > > should rather be extended when needed, implementing native > > encoder/decoder support for macOS specific attributes and additional > > payloads. This should then cover both, regular .pxar archives as well a= s > > split .mpxar/ppxar archives, while for the latter, the size of .mpxar i= s > > ideally kept small, placing larger payloads into the .ppxar file. > > Ideally the .mpxar only contains metadata needed for file change > > detection between subsequent backup runs and the data needed for fast > > content introspection, only fetching .ppxar content chunks when > > required, keeping metadata lookups fast. > > > > Since this will require a new pxar format version, ideally it would be > > extended in a manner which covers windows filesystem attributes as well= , > > so no further format changes are required for that. Backwards > > compatibility must be maintained, the newer implementation fully capabl= e > > of encoding and decoding the previous format versions. > > > Native archive/restore tests cover Unicode filenames, symlinks, > > > hardlinks, binary xattrs, FinderInfo, a resource fork larger than 1 > > > MiB, nanosecond times and destination name alias rejection before > > > truncation. A fresh Homebrew source installation on Apple Silicon als= o > > > passed backup and restore against a dedicated PBS test namespace, bot= h > > > unencrypted and client-side encrypted. Repeated backups reused 100% o= f > > > primary and auxiliary archive data. Linux client compilation and > > > format regression checks accompany the port. > > > > > > The work also separates client formats from Linux server dependencies > > > and fixes native fd/xattr/time/UUID APIs and mmap alignment on 16 KiB > > > pages. The inherited endian_trait ownership issue (RUSTSEC-2021-0039) > > > is addressed by a documented local MIT-licensed compatibility fork > > > using panic-safe array mapping and Copy-bound slices; it is not > > > presented as a published upstream fix. > > > > Note that while PBS depends on endian_trait, there is currently no > > codepath which does implement the Endian trait directly and would lead > > to a panic. The dependency is however already been flagged internally a= s > > to be phased out. > > > > > Current scope and limits: Apple Silicon is tested; Intel is not yet > > > tested. FUSE mount/map remain Linux-only. There is no automatic iClou= d > > > placeholder download or atomic APFS snapshot. Protected system-manage= d > > > xattrs can produce explicit restore warnings. This is an experimental > > > prototype, not a claim of official support or a merge-ready packaging > > > series. > > > > > > The implementation was developed with extensive AI assistance. Its > > > provenance, source patches and actual test results are available for > > > review. Proxmox confirmed on 26 October 2026 that my Individual CLA i= s > > > on record for future contributions. No independent human code review > > > is claimed at this stage. > > > > > > Before preparing a cleaned-up cross-repository patch series, would > > > this native client scope and metadata representation fit the upstream > > > architecture? I would welcome guidance on a preferred representation, > > > required review and how to split the > > > proxmox-backup/proxmox/pxar/pathpatterns changes. > > > > Patches have to be send to the developers mailing list for review > > following the guidelines as outlined in > > https://pbs.proxmox.com/wiki/Developer_Documentation. The patch series > > can span multiple repositories, the patches subject prefixed according > > to the repositories they belong to. > > > > Best regards, > > Chris