[pve-devel] [PATCH container/docs/lxc/manager/proxmox{, -perl-rs}/storage v4 00/15] support OCI images as container templates

[Filip Schauer <f.schauer@proxmox.com>]

Add basic support for OCI (Open Container Initiative) images [0] as
container templates.

An OCI image can be obtained from a registry like Docker Hub. This patch
series does not implement the OCI Distribution Spec, so this requires
external tools.

Either using Docker:

```
$ docker pull httpd
$ docker save httpd > httpd.tar
```

Or using Podman:
When using Podman, the format needs to be explicitly specified,
otherwise it defaults to docker-archive.

```
$ podman pull httpd
$ podman save --format=oci-archive httpd > httpd.tar
```

Or using skopeo:

```
$ skopeo copy docker://httpd:latest oci-archive:httpd.tar:latest
```

The tarball can be uploaded to a storage as a container template and
then used during container creation. It is automatically detected that
the container template is an OCI image. The resulting container still
uses the existing LXC framework.

# Dependencies:

To be able to build `proxmox-oci`, the `oci-spec` crate is required as a
dependency. A patch from Christoph [1] packages the `oci-spec` crate as
a deb package. Alternatively if the `oci-spec` crate is not yet
packaged, it can be downloaded from crates.io.

Here is a little script to download the `oci-spec` crate along with its
dependencies:

```sh
download_crate() {
    CRATE_NAME=$1
    CRATE_VERSION=$2
    CRATE_SHA256=$3

    wget https://crates.io/api/v1/crates/$CRATE_NAME/$CRATE_VERSION/download

    COMPUTED_SHA256=$(sha256sum download | awk '{ print $1 }')
    if [ "$COMPUTED_SHA256" != "$CRATE_SHA256" ]; then
        echo "Checksum mismatch"; exit 1
    fi

    tar -xf download
    rm download
    mv $CRATE_NAME-$CRATE_VERSION /usr/share/cargo/registry/
    echo "{\"package\":\"$CRATE_SHA256\",\"files\":{}}" > /usr/share/cargo/registry/$CRATE_NAME-$CRATE_VERSION/.cargo-checksum.json
}

download_crate strsim 0.11.1 7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f
download_crate ident_case 1.0.1 b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39
download_crate darling_macro 0.20.11 fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead
download_crate darling_core 0.20.11 0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e
download_crate darling 0.20.11 fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee
download_crate proc-macro-error-attr2 2.0.0 96de42df36bb9bba5542fe9f1a054b8cc87e172759a1868aa05c1f3acc89dfc5
download_crate derive_builder_core 0.20.2 2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8
download_crate thiserror-impl 2.0.0 22efd00f33f93fa62848a7cab956c3d38c8d43095efda1decfc2b3a5dc0b8972
download_crate rustversion 1.0.20 eded382c5f5f786b989652c49544c4877d9f015cc22e145a5ea8ea66c2921cd2
download_crate heck 0.5.0 2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea
download_crate proc-macro-error2 2.0.1 11ec05c52be0a07b08061f7dd003e7d7092e0472bc731b4af7bb1ef876109802
download_crate derive_builder_macro 0.20.2 ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c
download_crate thiserror 2.0.0 15291287e9bff1bc6f9ff3409ed9af665bec7a5fc8ac079ea96be07bca0e2668
download_crate strum_macros 0.27.1 c77a8c5abcaf0f9ce05d62342b7d298c346515365c36b673df4ebe3ced01fde8
download_crate strum 0.27.1 f64def088c51c9510a8579e3c5d67c65349dcf755e5479ad3d010aa6454e2c32
download_crate getset 0.1.5 f3586f256131df87204eb733da72e3d3eb4f343c639f4b7be279ac7c48baeafe
download_crate derive_builder 0.20.2 507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947
download_crate oci-spec 0.8.1 57e9beda9d92fac7bf4904c34c83340ef1024159faee67179a04e0277523da33
```

Since librust-oci-spec-dev is in the proxmox-oci/debian/control file, a
dummy package needs to be installed, so dpkg-checkbuilddeps does not
complain.

dummy_librust_oci_spec.equivs:

```
Package: librust-oci-spec-dev
Version: 0.8.1
Provides: librust-oci-spec-0.8+default-dev (= 0.8.1-1)
```

```
$ equivs-build dummy_librust_oci_spec.equivs
$ dpkg -i ./librust-oci-spec-dev_0.8.1_all.deb
```

# Build & install order:

OCI image support:
1. proxmox
2. proxmox-perl-rs
3. lxc
4. pve-container

.tar container template support:
1. pve-storage
2. pve-manager

In no particular order:
* pve-docs

[0] https://github.com/opencontainers/image-spec/blob/main/spec.md
[1] https://lore.proxmox.com/pve-devel/20250606103719.533030-2-c.heiss@proxmox.com/

Changed since v3:
* proxmox: add rustdoc comments
* proxmox: add unit tests
* proxmox: correctly handle OCI image whiteout edge cases
* proxmox/proxmox-perl-rs: parse_and_extract_image: add argument for
  CPU architecture
* proxmox-perl-rs: improve rustdoc comments
* lxc: conf: split `lxc.environment` into `runtime` and `hooks`
* pve-container: pass environment variables to container via
  `lxc.container.runtime`

Changed since v2:
* lxcfs: lxc.mount.hook: override env variables from container config
* pve-container: rebase onto newest master (5a8b3f962f16) and re-format
  with proxmox-perltidy
* pve-container: check whether archive is an OCI image before trying to
  parse it as one
* pve-container: add an "ipmanagehost" property to pct.conf to indicate
  that network interface IP configuration should be handled by the host.
* pve-container: manage_dhclient: add a FIXME comment regarding the
  AppArmor profile: "use a profile that confines writes to
  /var/lib/lxc/$vmid and rootfs"
* pve-container: kill_dhclients: untaint pid from pidfile
* pve-container: fix manage_dhclient called with 'stop' instead of
  'start' for IPv6 when container is started
* pve-docs: add OCI image docs
* proxmox-perl-rs: rebase onto newest master (3809f1229602)
* proxmox-perl-rs: forward all errors to Perl
* proxmox-perl-rs: remove oci-spec dependency
* pve-manager: rebase onto newest master (84b22751f211) and re-format
* proxmox: io: introduce RangeReader for bounded reads
* proxmox: oci: remove reachable unwraps & refactor code
* proxmox: oci: increase hasher buffer size from 4096 to 32768 (matching
  internal sha2::Digest buffering)
* proxmox: oci: preserve permissions and xattrs during rootfs extraction
* proxmox: oci: handle whiteouts & opaque whiteouts
* pve-storage: Modify VZTMPL_EXT_RE_1 regex to put "tar" into capture
  group when matching on a .tar file.

Changed since v1:
* Fix entrypoint command missing Cmd
* Set lxc.signal.halt according to StopSignal (Fixes container shutdown)
* setup: Ensure that both /etc/systemd/network and
  /etc/systemd/system-preset exist before writing files into them.
* ui: storage upload: accept *.tar files as vztmpl
* proxmox-perl-rs: rebase on latest master (3d9806cb3c7f)
* proxmox-perl-rs: add new dependencies to debian/control
* proxmox-oci: refactor errors and use `thiserror` to avoid boilerplate

proxmox:

Filip Schauer (3):
  io: introduce RangeReader for bounded reads
  add proxmox-oci crate
  proxmox-oci: add tests for whiteout handling

 Cargo.toml                                    |   1 +
 proxmox-io/src/lib.rs                         |   3 +
 proxmox-io/src/range_reader.rs                | 175 ++++++++++
 proxmox-oci/Cargo.toml                        |  25 ++
 proxmox-oci/debian/changelog                  |   5 +
 proxmox-oci/debian/control                    |  45 +++
 proxmox-oci/debian/debcargo.toml              |   7 +
 proxmox-oci/src/lib.rs                        | 324 ++++++++++++++++++
 proxmox-oci/src/oci_tar_image.rs              | 144 ++++++++
 proxmox-oci/tests/extract_whiteouts.rs        |  92 +++++
 .../oci_test_whiteout_current_directory.tar   | Bin 0 -> 8704 bytes
 .../oci_test_whiteout_dead_symlink_parent.tar | Bin 0 -> 8704 bytes
 .../oci_test_whiteout_root_breakout.tar       | Bin 0 -> 7168 bytes
 ...oci_test_whiteout_root_parent_breakout.tar | Bin 0 -> 7168 bytes
 .../oci_test_whiteout_symlink.tar             | Bin 0 -> 8704 bytes
 15 files changed, 821 insertions(+)
 create mode 100644 proxmox-io/src/range_reader.rs
 create mode 100644 proxmox-oci/Cargo.toml
 create mode 100644 proxmox-oci/debian/changelog
 create mode 100644 proxmox-oci/debian/control
 create mode 100644 proxmox-oci/debian/debcargo.toml
 create mode 100644 proxmox-oci/src/lib.rs
 create mode 100644 proxmox-oci/src/oci_tar_image.rs
 create mode 100644 proxmox-oci/tests/extract_whiteouts.rs
 create mode 100644 proxmox-oci/tests/oci_image_data/oci_test_whiteout_current_directory.tar
 create mode 100644 proxmox-oci/tests/oci_image_data/oci_test_whiteout_dead_symlink_parent.tar
 create mode 100644 proxmox-oci/tests/oci_image_data/oci_test_whiteout_root_breakout.tar
 create mode 100644 proxmox-oci/tests/oci_image_data/oci_test_whiteout_root_parent_breakout.tar
 create mode 100644 proxmox-oci/tests/oci_image_data/oci_test_whiteout_symlink.tar


proxmox-perl-rs:

Filip Schauer (1):
  add Perl mapping for OCI container image parser/extractor

 pve-rs/Cargo.toml          |  1 +
 pve-rs/Makefile            |  1 +
 pve-rs/debian/control      |  1 +
 pve-rs/src/bindings/mod.rs |  3 +++
 pve-rs/src/bindings/oci.rs | 29 +++++++++++++++++++++++++++++
 5 files changed, 35 insertions(+)
 create mode 100644 pve-rs/src/bindings/oci.rs


lxc:

Filip Schauer (1):
  lxc: conf: split `lxc.environment` into `runtime` and `hooks`

 ...c.environment-into-runtime-and-hooks.patch | 324 ++++++++++++++++++
 debian/patches/series                         |   1 +
 2 files changed, 325 insertions(+)
 create mode 100644 debian/patches/pve/0003-PVE-conf-split-lxc.environment-into-runtime-and-hooks.patch


pve-container:

Filip Schauer (7):
  config: add `lxc.environment.runtime`/`hooks`
  add support for OCI images as container templates
  config: add entrypoint parameter
  configure static IP in LXC config for custom entrypoint
  setup: debian: create /etc/network path if missing
  setup: recursively mkdir /etc/systemd/{network,system-preset}
  implement host-managed DHCP for containers with `ipmanagehost`

 src/PVE/API2/LXC.pm         |  98 +++++++++++++++++++++++++++++----
 src/PVE/LXC.pm              | 106 ++++++++++++++++++++++++++++++++++--
 src/PVE/LXC/Config.pm       |  33 ++++++++++-
 src/PVE/LXC/Setup/Base.pm   |   3 +-
 src/PVE/LXC/Setup/Debian.pm |   1 +
 5 files changed, 224 insertions(+), 17 deletions(-)


pve-storage:

Filip Schauer (1):
  allow .tar container templates

 src/PVE/Storage.pm        | 2 +-
 src/PVE/Storage/Plugin.pm | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


pve-manager:

Filip Schauer (1):
  ui: storage upload: accept *.tar files as vztmpl

 www/manager6/window/UploadToStorage.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


pve-docs:

Filip Schauer (1):
  ct: add OCI image docs

 pct.adoc | 72 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 8 deletions(-)


Summary over all repositories:
  31 files changed, 1472 insertions(+), 28 deletions(-)

-- 
Generated by git-murpp 0.6.0


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel