From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id D982E1FF0B6 for ; Thu, 12 Nov 2026 15:11:42 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 629B221662; Thu, 12 Nov 2026 15:11:38 +0100 (CET) From: Filip Schauer To: pve-devel@lists.proxmox.com Subject: [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Date: Thu, 12 Nov 2026 15:10:27 +0100 Message-ID: <20261112141109.137298-1-f.schauer@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794492678760 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.258 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 363DAZORAJGGFKFLIFPB36SDRDXG4FWL X-Message-ID-Hash: 363DAZORAJGGFKFLIFPB36SDRDXG4FWL X-MailFrom: f.schauer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is a first part of the work towards the OverlayFS integration for OCI images proposed in https://lore.proxmox.com/pve-devel/5709a436-fb72-4fa5-8bdc-702c3e96a5a7@proxmox.com/. # Background Currently, a pulled OCI image is a single tar and its layers are squashed into one rootfs volume per container. Shared base layers are therefore duplicated per container and redownloaded on every pull. The goal of the proposal linked above is to store layers once, content-addressed, in a new "oci" content type on directory-based storages, and to mount them read-only as OverlayFS lowerdirs with a thin writable upperdir per container. The groundwork functions in this series use the following storage layout, which deviates slightly from the original RFC proposal: ``` oci/ +-- layers/ | +-- sha256-111.../ | +-- sha256-222.../ | `-- by-blob-id/ | +-- sha256-abc... | `-- sha256-def... `-- sha256-9f86d081... # image config, named by its ImageID ``` # Main points of this series: 1. A native OCI registry client in the proxmox-oci crate, which replaces the external skopeo binary. 2. Groundwork for a future "oci" content type: functions to pull an image into a layer-deduplicating storage layout, remove images from it and garbage-collect unused layers. These are exposed in PVE::RS::OCI, but nothing in this series calls them yet. 3. fix #7058: Support for Docker Image Manifest V2 media types # Build & install order: 1. proxmox 2. proxmox-perl-rs 3. pve-container, pve-manager, pve-storage # Version and dependency bumps 1. proxmox: * bump proxmox-oci 2. proxmox-perl-rs: * depend on the newly bumped proxmox-oci * bump libpve-rs-perl (Since it renamed parse_and_extract_image, it also needs `Breaks: pve-container (<< x.y.z)`. 3. pve-container, pve-manager, pve-storage: * depend on the newly bumped libpve-rs-perl The bumps themselves are not part of this series. proxmox: Filip Schauer (15): oci: rename error when no manifest can be selected oci: add `os` parameter to parse_and_extract_image oci: add a digest helper module oci: reduce redundant layer decompression during extraction oci: factor out parsing and extraction into modules oci: explicitly set mtime of directories when extracting layer oci: add a registry client oci: add a public list_repository_tags function oci: rename parse_and_extract_image to be tar-specific oci: rename ProxmoxOciError to ParseAndExtractTarError oci: add a public pull_image_as_tar function oci: add a public pull_image_to_oci_storage function fix #7058: oci: add compatibility for Docker Image Manifest V2 oci: include media type in ParseError::WrongMediaType oci: add public functions for removing OCI images proxmox-oci/Cargo.toml | 13 +- proxmox-oci/debian/control | 24 +- proxmox-oci/src/client.rs | 622 ++++++++++++++++++++++++++ proxmox-oci/src/digest.rs | 113 +++++ proxmox-oci/src/extract.rs | 343 +++++++++++++++ proxmox-oci/src/lib.rs | 722 +++++++++++++++++-------------- proxmox-oci/src/oci_tar_image.rs | 12 +- proxmox-oci/src/parse.rs | 57 +++ proxmox-oci/tests/helper/mod.rs | 11 +- 9 files changed, 1578 insertions(+), 339 deletions(-) create mode 100644 proxmox-oci/src/client.rs create mode 100644 proxmox-oci/src/digest.rs create mode 100644 proxmox-oci/src/extract.rs create mode 100644 proxmox-oci/src/parse.rs proxmox-perl-rs: Filip Schauer (7): pve-rs: oci: update doc comment of parse_and_extract_image pve-rs: oci: add `os` parameter to parse_and_extract_image pve-rs: oci: expose list_repository_tags pve-rs: oci: rename parse_and_extract_image to be tar-specific pve-rs: oci: expose pull_image_as_tar pve-rs: oci: expose pull_image_to_oci_storage pve-rs: oci: expose functions for removing OCI images pve-rs/src/bindings/oci.rs | 115 +++++++++++++++++++++++++++++++++++-- 1 file changed, 109 insertions(+), 6 deletions(-) pve-container: Filip Schauer (1): oci create: rename parse_and_extract_image to be tar-specific src/PVE/LXC/Create.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) pve-manager: Filip Schauer (1): use proxmox-oci instead of skopeo for listing tags from OCI registries PVE/API2/Nodes.pm | 13 ++----------- debian/control | 1 - 2 files changed, 2 insertions(+), 12 deletions(-) pve-storage: Filip Schauer (2): use proxmox-oci instead of skopeo for pulling from OCI registries api: oci image pull: fix temp file cleanup on task stop debian/control | 1 - src/PVE/API2/Storage/Status.pm | 22 ++++++++-------------- 2 files changed, 8 insertions(+), 15 deletions(-) Summary over all repositories: 15 files changed, 1699 insertions(+), 374 deletions(-) -- Generated by murpp 0.12.1