From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C1EF51FF0E4 for ; Tue, 14 Jul 2026 15:19:16 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BA867213EB; Tue, 14 Jul 2026 15:19:15 +0200 (CEST) Message-ID: <5709a436-fb72-4fa5-8bdc-702c3e96a5a7@proxmox.com> Date: Tue, 14 Jul 2026 15:19:09 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Proxmox VE development discussion From: Filip Schauer Subject: [RFC container/storage] OCI image OverlayFS integration proposal Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1784035133759 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.350 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: HEWQ6JGYIW75XARLTNOZSLIICPTFSDGU X-Message-ID-Hash: HEWQ6JGYIW75XARLTNOZSLIICPTFSDGU X-MailFrom: f.schauer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: I would like to propose extending the OCI container tech preview with OverlayFS, so that image layers are stored once (content-addressed & deduplicated) and mounted read-only as lowerdirs, with a thin writable upperdir per container, rather than being squashed into one rootfs volume per container. --- ## Status Quo In the current technology preview of OCI (Open Container Initiative) integration, OCI images can be used as templates for LXC containers. They can be pulled from an OCI registry into a storage that supports the "vztmpl" (CT Templates) content type via the "Pull from OCI Registry" button in the web UI. A downloaded OCI image is squashed into a single TAR archive containing the config and all layers. When creating a container, an OCI image can be selected as the "ostemplate". It is automatically detected as an OCI template and its layers are extracted and squashed into a single container volume (stored in the "rootdir" content type) used as the container's rootfs. The container is also configured according to the config supplied in the OCI image. Once the container is created, it works just like any other LXC container. The OCI image was effectively used as a template for creating a container that fits into the existing LXC container framework. There is no separate OCI runtime (e.g. containerd) being used here. ## Problem Because all layers are currently squashed into one container disk, there is no deduplication. When multiple containers are created from the same base image, all base layers are always duplicated into each new container rootfs. Furthermore, multiple OCI images might contain identical layers. Since each OCI image bundles its layers into one TAR archive, the layers are not deduplicated between OCI images. Additionally, when pulling from a registry an OCI image that contains layers already stored locally, those layers are redundantly re-downloaded. ## Objective Deduplicate shared layers across containers and across images via OverlayFS. When pulling from a registry, only download layers not already stored locally. Images are immutable, content-addressed, and referenced by digest, so a container's rootfs can never be silently rebased by re-pulling a tag. An administrator who deliberately wants to rebase a container onto a newer image can still do so explicitly, by editing the container config and pointing it at a different digest. (This has the potential to break a container, especially if a manual `apt upgrade` was performed in the container and then the base image is swapped out.) # Proposal: OverlayFS integration for Proxmox VE An independent content type just for OCI images, with the container rootfs remaining an ordinary "rootdir" volume on any storage, referencing the underlying image via a new property in the container config. ## Storage layout of new content type "oci" Add a new content type, "oci", available only on directory-based storages (the same kind of restriction already applied to "vztmpl"/"backup"/"snippets"). Volume naming: `:oci/sha256-`, where the digest is the image's own OCI ImageID (the hash of its image configuration JSON). Directory structure of the content type: ``` oci/ +-- layers/ |   +-- sha256-111.../ |   +-- sha256-222.../ |   `-- by-blob-id/ |       +-- sha256-abc... |       `-- sha256-def... `-- images/     +-- sha256-9f86d081...     `-- sha256-2c26b46b... ``` ### `layers/` Stores the extracted contents of the immutable lower layers. Whiteouts are stored as Linux-native whiteout nodes. Each layer has its own directory named after its uncompressed layer archive digest (Diff ID). These directories are strictly read-only. `by-blob-id/` caches which blob digest corresponds to which locally stored layer. This is needed for lookups to avoid redundant redownloading of layers. An image manifest only supplies the digests of the potentially compressed layer blobs. But under `layers/` the names of the directories only hold the uncompressed digest. So this `by-blob-id/` directory correlates compressed digests to uncompressed digests, which lets us avoid redundant downloads of already stored layers. It contains text files named after a layer's compressed blob digest. The content of the file is simply the uncompressed `diff_id` string. This maps a compressed blob hash to the local uncompressed layer directory. Layers are never independently addressable volumes and are never returned by volume-listing operations. They are internal bookkeeping, comparable to how a thin-pool plugin manages internal block sharing without exposing individual chunks as volumes. ### `images/` Each OCI image gets its own JSON file, which is simply its corresponding OCI image configuration. The name of the file is its SHA256 hash, which is also the ImageID by the OCI spec. This file originates directly from the OCI image. It holds parameters like CPU architecture, entrypoint, environment variables... and the `rootfs.diff_ids` array specifying the ordered stack of layers in `layers/`. ## Pulling an image from a registry 1. Fetch the image index from the registry. 2. Find & Fetch the image manifest corresponding to the correct CPU    architecture. 3. Fetch the image configuration referenced by the manifest. 4. Check `layers/by-blob-id/` for every layer listed    in the manifest, to determine which layers are already stored    locally. 5. Download only the missing layers. Decompress, verify the digest, and    extract each layer tarball into `layers//`. 6. Create a `layers/by-blob-id/` text file containing    the uncompressed digest string for every layer. Replace whiteout    files with actual whiteout nodes. 7. Save the image configuration under `images/`. ## Container configuration & container rootfs disks The container rootfs remains an ordinary "rootdir" volume. The only addition is a new `ocibase` property on the "rootfs" mountpoint: a reference to an underlying OCI image volume. This property must be accepted only on the "rootfs" mountpoint, never on other mountpoints. The image storage referenced by this property can be on a different storage than the rootfs volume itself. The property must be supplied when the "rootfs" volume has an `overlay` suffix (`vm--overlay` instead of `vm--disk-`) in its name, similar to how the `cloudinit` suffix for VM disks (`vm--cloudinit` instead of `vm--disk-`) does not allow the `iothread` and `ro` properties. On the other hand, the `ocibase` property must not be supplied when the disk does not have an `overlay` suffix. Other than the regular container rootfs disks that contain the container file system directly, an `overlay` disk contains two directories at its root: ``` upper/  # Writable topmost layer of the container rootfs work/   # Empty directory needed by OverlayFS. OverlayFS needs this to be on the same file system as `upper/`. ``` The `overlay` suffix allows us to distinguish between an `overlay` disk with said layout and a regular container disk with the file system placed directly at its root. ## Creating a container 1. Allocate a new volume in the `rootdir` content type containing:     * empty `upper/` directory     * empty `work/` directory 2. Point `rootfs` in the container config to this disk along with the    `ocibase` property pointing to the OCI image volume. ## Starting the container 1. If the `rootfs` in the container config points to a disk with the    `overlay` suffix, look for the `ocibase` property to identify the    base image. 2. On the OCI image storage, read `images/sha256-` to    extract the `rootfs.diff_ids` array. 3. Map the `diff_ids` to their absolute paths in the `layers/` directory    to generate the `lowerdir` string in stack order. 4. Mount the overlay filesystem.    (Combining OverlayFS with idmapping would probably require  https://lore.kernel.org/all/20260615-work-idmapped-overlayfs-v1-0-7381632aa402@kernel.org/    to be applied in the kernel.)    Very simplified example command for intuition:    ```    mount -t overlay overlay --map-users 0:100000:65536 --map-groups 0:100000:65536 -o lowerdir=/var/lib/vz/oci/layers/sha256-111...:/var/lib/vz/oci/layers/sha256-222...,upperdir=/path/to/mountdir/of/vm-100-overlay/upper,workdir=/path/to/mountdir/of/vm-100-overlay/work /var/lib/lxc/100/rootfs    ``` ## Cloning a container This requires no changes. Since the rootfs remains an ordinary volume and the new property is an ordinary scalar config value, existing clone/copy logic already copies both correctly. ## Deleting an image Because layers are deduplicated and shared, deleting a base image requires a reference-counting mechanism: 1. Iterate through all container configs to verify that no `rootfs`    references the image in the `ocibase` property (including pending    changes and snapshots). Abort if any do. 2. Remove the target image configuration file in `images/`. 3. For every `diff_id` that was present in the deleted image config,    check if any remaining config across all other stored `images/` still    references it. 4. For any `diff_id` without references:     * Delete its directory from `layers/`.     * Delete all corresponding mapping files in `layers/by-blob-id/`. ## Open Questions & Notes * Backups and migration will require special handling because container   root filesystems are no longer fully self-contained. For migration   probably require the image to already be present on the target. For   backups I am unsure whether we should include the image aswell. * Do all directory-based storage plugins support whiteout nodes? * Human-readable image tags --- I will follow up with patches once I have a working prototype.