From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 149D71FF0B7 for ; Fri, 30 Oct 2026 22:14:45 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C8B84215F1; Fri, 30 Oct 2026 22:14:44 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; t=1793394873; cv=none; d=google.com; s=arc-20260327; b=aGiqV3e+x0Dbi4TKGP/veKEYLhhNqcIgByXYdbaBZaOMmEYoIFtR7cLRqCtjlONFqp SuKfRUwo15Ze1cRd428q+aDgdpuAJcccGcKbA4Le8VCDbCqQK+Gg4VF2X9c7PfbR+Df2 wqdTjbYXZIh451ge8xwOKcXrwG0WimP8D85BFFcTDhvqsjGUdSfGm+6LUszIue/HrktI R2hZx96mwbIJ3Wp9f0WB/vxwXNv6ufPNwyycCe7r9RwERylAhAdqvQBl6pbeemX8w0GW OkzV+kk/UnwJriiAOKMxBPcqfoIdRHjDyge8D9BkcahPPZvDvrYu+iX4DUVIeZ1b+TaQ 9kuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:cc:to:subject:message-id:date:references :in-reply-to:from:mime-version:dkim-signature; bh=JvDRlX6HFT8mwnk+VEjtYx0I5IuBz7EkbfmkZC8N9cs=; fh=x735zFtK7d+S3igtQIHSkrR20uCSKtnfpmFHdvkFDH0=; b=Mjw1k7cABQCFXr5eJpdp01wNU4renTg0IM2iKCWw3eod8qUdz8huO0DNBcaqbaIXbx xnu4atstvYZul8Wws7tHOrwMrGvKzuiq3ES89OzqEdOy0kEgPzPazxI6tY6jPLL1Cf/R mr1V6XKXHidRvo2kv2fL+Ga/132GYOtslD3XjKtmPCYur/eBx5QtyMRdKq/a1T2EibL9 IJ/ZK/oc85L1avN/3/6fTIuajkdZOoZO9wbKG55e/vT+XT3WdA7RALW22fdqgDuROYih KohoO7JEX3e9jBwVCMqKchoUzHKpJa2zdoQrTu8vogmOlTCalzG9XGjprmU8lQBn7Fu2 RAkQ==; darn=lists.proxmox.com ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1793394873; x=1793999673; darn=lists.proxmox.com; h=content-transfer-encoding:content-type:cc:to:subject:message-id :date:references:in-reply-to:from:mime-version:from:to:cc:subject :date:message-id:reply-to:content-type; bh=JvDRlX6HFT8mwnk+VEjtYx0I5IuBz7EkbfmkZC8N9cs=; b=WibO0P4FJrQ1zX1hvbOKsnCT2v4S7WdXQxpvcKQf83HcbEuYbtORJUU1f/CLOwp4j2 92agE9wDsc63UZaC+ZyuinsNko25XEu1TlmdB1gbfyXKbxBaHqPBslZSnE5yRi135U+V YrvRvzCdCRBCXyd5OuOJ8I/BZ1N7KOXTgSrxxnH4xk9O5l2L+8RWu2xPUVqlkxESXpRj ydsm/piXD5Zvo2ff+QaD0yA0vgSVMzXK0Go/xardA5RIexDKyPB5cBvZgpoVAtLDJDER rZOA02Zhg1xoM+TFxiNeVTuY6nmbMtekuOorz5Ld6YxUsLo8EBy5cM7V7LJ/xeRGyXQi bFyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793394873; x=1793999673; h=content-transfer-encoding:content-type:cc:to:subject:message-id :date:references:in-reply-to:from:mime-version:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=JvDRlX6HFT8mwnk+VEjtYx0I5IuBz7EkbfmkZC8N9cs=; b=VUwXiTEgh/ad/5xdf9vGf9rTux2SbXeQs/zNaIJHxP3x+V+w91BPBP4ZkljU5nQXmY TqvtpU25zLIaef1SnSxK5oTo2LkyEztLkatssNgTYhC/xAPrdRF3fhfAadz7Oex/sta6 ug4/X2aXRQbL21lwqtqr4Oo9y9QTQpc/8fA/6nAqZbHMq2ATSNcgqH+WHUQJfzBIqqzI ndQUjjLGC0f9Wnrylw6bQqx71bA8OHsrJO2tIr0AVN4gKJZ/w3yVIzkf1e57lh3Xzxs7 DgsJzTCjx/WlwnIGJNaMc4ueLyZeyq3oTTjRtAZtU7Rcye/bojPmPLQImPJ9jiyJjTXe YOmQ== X-Gm-Message-State: AFq9FYJrJBK6/czu4W5uQYlBth5GGeZQVyaDAE3LdK0yq9jhstB1/xCT J/4qSG4Ezxgx+Zl6E+R7A2S/i9MsmcQ04+11QNDjX5p6ED05Xb6F674OuC6//DLV9eafkNi4+ky hFBxEYoOnkfdRaqfvAhOUjnEXaDFZDKMWf4a7 X-Gm-Gg: AYBFou1u/eT8N/j5uqX+w8JeMEcOVU4mwcZjVSepeqmK5N2C1HqEevasUkiv5gkSVXw 2kFBTdYVPJERjpjRxRKt4zL75Kz2+z4em6n7I3Nf1DsU+nrG62yK9FyAW32cxqCMFU8dQCSBC7k wjrCiL9I+wuD8MJZQYCm62og7Wj0NSb4im9mWjxpDlUoFW2xDfmRaa4SHwZyLa/fh0l86qNkyZc u18vEYEa83rPKbfBtBJuW8mR//Ua4VGyr5Ikx95bUPFy79gXB3trFBy538cKyGT81Gw8jPKsVcS UOESe/ESPTymsS9bC+dm0dqK5Ug6w9msjS90iYpVY6J8NxoCJT0AIEZcyX6OAc2jQ6kmntyucHO BhtEgtk5K4eXepTUeWhntyJeLwLISqKrZ6B8sz0Q8CyaAt5/PmBCvGh5TCrrmtebFr1E= X-Received: by 2002:aa7:c697:0:b0:6b2:2348:7772 with SMTP id 4fb4d7f45d1cf-6b4d4686edamr1891367a12.8.1793394872914; Fri, 30 Oct 2026 14:14:32 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?Q?Adam_Oko=C5=84?= In-Reply-To: References: Date: Fri, 30 Oct 2026 16:14:32 -0500 X-Gm-Features: AclHuK8NmReXUiu-5IdoxJLrHOgSmftpqYAvYjiHWxtnwMZ_lMDf6v2w3V6ws8A Message-ID: Subject: Re: [RFC] Native macOS PBS client: filesystem metadata and tested prototype To: Christian Ebner Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.375 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FORGED_GMAIL_RCVD 1 'From' gmail.com does not match 'Received' headers FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NAUIUQEUQ6UHPURQJWMCA47CWQA6K24P X-Message-ID-Hash: NAUIUQEUQ6UHPURQJWMCA47CWQA6K24P X-MailFrom: adamokon@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pbs-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Chris, Thank you for the architectural guidance. I have implemented an experimental revision following the native PXAR approach you suggested. The changes relative to the initial macOS prototype are available here: https://github.com/adamokon/proxmox-backup-macos/pull/2 The wire layout and review boundaries are documented here: https://github.com/adamokon/proxmox-backup-macos/blob/codex/native-pxar-met= adata/docs/native-pxar-draft.md The revised implementation adds native metadata, attribute descriptor and payload records to the PXAR encoder/decoder. It supports both unified .pxar archives and split .mpxar/.ppxar archives. In the split representation, descriptors, lengths and content digests remain in .mpxar, while attribute values and resource forks are streamed into .ppxar. Metadata-only traversal does not require the payload stream, and selective restoration can fetch the native payloads for the selected entries. New backups no longer use the synthetic xattr envelope, auxiliary archive or metadata map from the initial prototype. The new reader retains support for that prototype's backups, and the previous PXAR v1/v2 encoding and decoding paths remain available. For Darwin, the records cover birthtime, BSD flags, ordered UUID ACL entries and native xattrs/resource forks. The draft also represents Windows file attributes, FILETIMEs, security descriptors, reparse data and named streams. Windows coverage is currently synthetic wire-format testing; Windows filesystem capture/restore APIs have not been implemented or tested. Validation completed for this revision: - 16 isolated Apple Silicon tests passed, including actual unified/split filesystem restoration, selective subdirectory restoration, metadata-only and random access, malformed/truncated/corrupt inputs, and the original upstream old-timestamp compatibility fixture. - Linux CI passed: PBS format regressions, PXAR library and native wire tests, path matching, endian/mmap checks and full Linux client compilation: https://github.com/adamokon/proxmox-backup-macos/actions/runs/41128035455 - Encrypted backup and restore against a real PBS server passed for the legacy prototype and the new unified/split representations. Comparisons covered file bytes, hardlinks, symlink targets, permissions, timestamps, BSD flags, ACLs, binary xattrs, FinderInfo and a resource fork larger than 1 MiB. All five final test snapshots were independently confirmed on PBS with verification state "ok". - In the split fixture, .mpxar was 1779 bytes for approximately 5 MiB of payloads. Repeated unified/split backups reused 100% of archive data chunks. The draft uses deliberately unallocated experimental version/record identifiers, pending upstream agreement. One performance limitation remains: files carrying native attributes are conservatively re-read/re-encoded rather than eliding reads through split payload-range reuse. Chunk deduplication remains effective, but that reuse optimization needs further work. The implementation continues to use extensive AI assistance and still requires independent human review. Intel macOS has not been tested. Thank you also for clarifying the current endian_trait usage and the planned dependency removal; the inherited compatibility fork should be considered separately from the native format proposal. Could you comment on whether this revised framing and payload placement fit the intended PXAR architecture? In particular, are the proposed platform records and Windows metadata categories an appropriate starting point, and what would you change before preparing a cleaned-up, cross-repository patch series for the mailing list? Best regards, Adam Oko=C5=84 On Thu, 29 Oct 2026 11:05:46 +0100, Christian Ebner w= rote: > Hi, > > On 10/27/26 10:31 AM, Adam Oko=C5=84 wrote: > > Hello, > > > > I would appreciate architectural feedback on a native macOS port of > > proxmox-backup-client, based on PBS v4.2.8. The prototype runs without > > Docker or a Linux guest and keeps the usual backup/restore CLI, PBS > > chunking, deduplication and client-side encryption. > > > > Source and review workspace: > > https://github.com/adamokon/proxmox-backup-macos/pull/1 > > thanks for reaching out. As an upfront disclaimer: please note that we > can only accept high quality code contributions which keep the codebase > stable and maintainable. > > The main design question is preservation of Darwin filesystem > > metadata. The prototype uses a bounded, versioned envelope in standard > > PXAR xattr records to describe ordered UUID ACLs, birthtime, BSD flags > > and native attribute content references. Attribute bytes, including > > large resource forks, are content-addressed regular files in an > > additional standard PXAR archive in the same snapshot. A > > manifest-listed map associates it with the primary archives. The > > server protocol and PXAR record format are unchanged. Temporary > > spooling is authenticated-encrypted with an ephemeral local key, then > > the ordinary PBS encryption and deduplication pipeline processes the > > upload stream. > > While not being an expert when it comes to macOS and it's file > attributes, I don't think the chosen initial implementation approach is > the best fitting one. > > Rather than trying to bypass changes to the pxar archive format, it > should rather be extended when needed, implementing native > encoder/decoder support for macOS specific attributes and additional > payloads. This should then cover both, regular .pxar archives as well as > split .mpxar/ppxar archives, while for the latter, the size of .mpxar is > ideally kept small, placing larger payloads into the .ppxar file. > Ideally the .mpxar only contains metadata needed for file change > detection between subsequent backup runs and the data needed for fast > content introspection, only fetching .ppxar content chunks when > required, keeping metadata lookups fast. > > Since this will require a new pxar format version, ideally it would be > extended in a manner which covers windows filesystem attributes as well, > so no further format changes are required for that. Backwards > compatibility must be maintained, the newer implementation fully capable > of encoding and decoding the previous format versions. > > Native archive/restore tests cover Unicode filenames, symlinks, > > hardlinks, binary xattrs, FinderInfo, a resource fork larger than 1 > > MiB, nanosecond times and destination name alias rejection before > > truncation. A fresh Homebrew source installation on Apple Silicon also > > passed backup and restore against a dedicated PBS test namespace, both > > unencrypted and client-side encrypted. Repeated backups reused 100% of > > primary and auxiliary archive data. Linux client compilation and > > format regression checks accompany the port. > > > > The work also separates client formats from Linux server dependencies > > and fixes native fd/xattr/time/UUID APIs and mmap alignment on 16 KiB > > pages. The inherited endian_trait ownership issue (RUSTSEC-2021-0039) > > is addressed by a documented local MIT-licensed compatibility fork > > using panic-safe array mapping and Copy-bound slices; it is not > > presented as a published upstream fix. > > Note that while PBS depends on endian_trait, there is currently no > codepath which does implement the Endian trait directly and would lead > to a panic. The dependency is however already been flagged internally as > to be phased out. > > > Current scope and limits: Apple Silicon is tested; Intel is not yet > > tested. FUSE mount/map remain Linux-only. There is no automatic iCloud > > placeholder download or atomic APFS snapshot. Protected system-managed > > xattrs can produce explicit restore warnings. This is an experimental > > prototype, not a claim of official support or a merge-ready packaging > > series. > > > > The implementation was developed with extensive AI assistance. Its > > provenance, source patches and actual test results are available for > > review. Proxmox confirmed on 26 October 2026 that my Individual CLA is > > on record for future contributions. No independent human code review > > is claimed at this stage. > > > > Before preparing a cleaned-up cross-repository patch series, would > > this native client scope and metadata representation fit the upstream > > architecture? I would welcome guidance on a preferred representation, > > required review and how to split the > > proxmox-backup/proxmox/pxar/pathpatterns changes. > > Patches have to be send to the developers mailing list for review > following the guidelines as outlined in > https://pbs.proxmox.com/wiki/Developer_Documentation. The patch series > can span multiple repositories, the patches subject prefixed according > to the repositories they belong to. > > Best regards, > Chris