From: Filip Schauer <f.schauer@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS
Date: Thu, 12 Nov 2026 15:10:27 +0100 [thread overview]
Message-ID: <20261112141109.137298-1-f.schauer@proxmox.com> (raw)
This is a first part of the work towards the OverlayFS integration for
OCI images proposed in
https://lore.proxmox.com/pve-devel/5709a436-fb72-4fa5-8bdc-702c3e96a5a7@proxmox.com/.
# Background
Currently, a pulled OCI image is a single tar and its layers are
squashed into one rootfs volume per container. Shared base layers are
therefore duplicated per container and redownloaded on every pull. The
goal of the proposal linked above is to store layers once,
content-addressed, in a new "oci" content type on directory-based
storages, and to mount them read-only as OverlayFS lowerdirs with a thin
writable upperdir per container.
The groundwork functions in this series use the following storage
layout, which deviates slightly from the original RFC proposal:
```
oci/
+-- layers/
| +-- sha256-111.../
| +-- sha256-222.../
| `-- by-blob-id/
| +-- sha256-abc...
| `-- sha256-def...
`-- sha256-9f86d081... # image config, named by its ImageID
```
# Main points of this series:
1. A native OCI registry client in the proxmox-oci crate, which replaces
the external skopeo binary.
2. Groundwork for a future "oci" content type: functions to pull an
image into a layer-deduplicating storage layout, remove images from
it and garbage-collect unused layers. These are exposed in
PVE::RS::OCI, but nothing in this series calls them yet.
3. fix #7058: Support for Docker Image Manifest V2 media types
# Build & install order:
1. proxmox
2. proxmox-perl-rs
3. pve-container, pve-manager, pve-storage
# Version and dependency bumps
1. proxmox:
* bump proxmox-oci
2. proxmox-perl-rs:
* depend on the newly bumped proxmox-oci
* bump libpve-rs-perl (Since it renamed parse_and_extract_image, it
also needs `Breaks: pve-container (<< x.y.z)`.
3. pve-container, pve-manager, pve-storage:
* depend on the newly bumped libpve-rs-perl
The bumps themselves are not part of this series.
proxmox:
Filip Schauer (15):
oci: rename error when no manifest can be selected
oci: add `os` parameter to parse_and_extract_image
oci: add a digest helper module
oci: reduce redundant layer decompression during extraction
oci: factor out parsing and extraction into modules
oci: explicitly set mtime of directories when extracting layer
oci: add a registry client
oci: add a public list_repository_tags function
oci: rename parse_and_extract_image to be tar-specific
oci: rename ProxmoxOciError to ParseAndExtractTarError
oci: add a public pull_image_as_tar function
oci: add a public pull_image_to_oci_storage function
fix #7058: oci: add compatibility for Docker Image Manifest V2
oci: include media type in ParseError::WrongMediaType
oci: add public functions for removing OCI images
proxmox-oci/Cargo.toml | 13 +-
proxmox-oci/debian/control | 24 +-
proxmox-oci/src/client.rs | 622 ++++++++++++++++++++++++++
proxmox-oci/src/digest.rs | 113 +++++
proxmox-oci/src/extract.rs | 343 +++++++++++++++
proxmox-oci/src/lib.rs | 722 +++++++++++++++++--------------
proxmox-oci/src/oci_tar_image.rs | 12 +-
proxmox-oci/src/parse.rs | 57 +++
proxmox-oci/tests/helper/mod.rs | 11 +-
9 files changed, 1578 insertions(+), 339 deletions(-)
create mode 100644 proxmox-oci/src/client.rs
create mode 100644 proxmox-oci/src/digest.rs
create mode 100644 proxmox-oci/src/extract.rs
create mode 100644 proxmox-oci/src/parse.rs
proxmox-perl-rs:
Filip Schauer (7):
pve-rs: oci: update doc comment of parse_and_extract_image
pve-rs: oci: add `os` parameter to parse_and_extract_image
pve-rs: oci: expose list_repository_tags
pve-rs: oci: rename parse_and_extract_image to be tar-specific
pve-rs: oci: expose pull_image_as_tar
pve-rs: oci: expose pull_image_to_oci_storage
pve-rs: oci: expose functions for removing OCI images
pve-rs/src/bindings/oci.rs | 115 +++++++++++++++++++++++++++++++++++--
1 file changed, 109 insertions(+), 6 deletions(-)
pve-container:
Filip Schauer (1):
oci create: rename parse_and_extract_image to be tar-specific
src/PVE/LXC/Create.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
pve-manager:
Filip Schauer (1):
use proxmox-oci instead of skopeo for listing tags from OCI registries
PVE/API2/Nodes.pm | 13 ++-----------
debian/control | 1 -
2 files changed, 2 insertions(+), 12 deletions(-)
pve-storage:
Filip Schauer (2):
use proxmox-oci instead of skopeo for pulling from OCI registries
api: oci image pull: fix temp file cleanup on task stop
debian/control | 1 -
src/PVE/API2/Storage/Status.pm | 22 ++++++++--------------
2 files changed, 8 insertions(+), 15 deletions(-)
Summary over all repositories:
15 files changed, 1699 insertions(+), 374 deletions(-)
--
Generated by murpp 0.12.1
next reply other threads:[~2026-11-12 14:11 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 14:10 Filip Schauer [this message]
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 01/26] pve-rs: oci: update doc comment of parse_and_extract_image Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 02/26] oci: rename error when no manifest can be selected Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 03/26] oci: add `os` parameter to parse_and_extract_image Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 04/26] pve-rs: " Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 05/26] oci: add a digest helper module Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 06/26] oci: reduce redundant layer decompression during extraction Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 07/26] oci: factor out parsing and extraction into modules Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 08/26] oci: explicitly set mtime of directories when extracting layer Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 09/26] oci: add a registry client Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 10/26] oci: add a public list_repository_tags function Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 11/26] pve-rs: oci: expose list_repository_tags Filip Schauer
2026-11-12 14:10 ` [PATCH manager 12/26] use proxmox-oci instead of skopeo for listing tags from OCI registries Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 13/26] oci: rename parse_and_extract_image to be tar-specific Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 14/26] pve-rs: " Filip Schauer
2026-11-12 14:10 ` [PATCH container 15/26] oci create: " Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 16/26] oci: rename ProxmoxOciError to ParseAndExtractTarError Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 17/26] oci: add a public pull_image_as_tar function Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 18/26] pve-rs: oci: expose pull_image_as_tar Filip Schauer
2026-11-12 14:10 ` [PATCH storage 19/26] use proxmox-oci instead of skopeo for pulling from OCI registries Filip Schauer
2026-11-12 14:10 ` [PATCH storage 20/26] api: oci image pull: fix temp file cleanup on task stop Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 21/26] oci: add a public pull_image_to_oci_storage function Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 22/26] pve-rs: oci: expose pull_image_to_oci_storage Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 23/26] fix #7058: oci: add compatibility for Docker Image Manifest V2 Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 24/26] oci: include media type in ParseError::WrongMediaType Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 25/26] oci: add public functions for removing OCI images Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 26/26] pve-rs: oci: expose " Filip Schauer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261112141109.137298-1-f.schauer@proxmox.com \
--to=f.schauer@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox