[RFC container/storage] OCI image OverlayFS integration proposal

[RFC container/storage] OCI image OverlayFS integration proposal

[Filip Schauer]

I would like to propose extending the OCI container tech preview with
OverlayFS, so that image layers are stored once (content-addressed &
deduplicated) and mounted read-only as lowerdirs, with a thin writable
upperdir per container, rather than being squashed into one rootfs
volume per container.

---

## Status Quo

In the current technology preview of OCI (Open Container Initiative)
integration, OCI images can be used as templates for LXC containers.
They can be pulled from an OCI registry into a storage that supports the
"vztmpl" (CT Templates) content type via the "Pull from OCI Registry"
button in the web UI. A downloaded OCI image is squashed into a single
TAR archive containing the config and all layers. When creating a
container, an OCI image can be selected as the "ostemplate". It is
automatically detected as an OCI template and its layers are extracted
and squashed into a single container volume (stored in the "rootdir"
content type) used as the container's rootfs. The container is also
configured according to the config supplied in the OCI image. Once the
container is created, it works just like any other LXC container. The
OCI image was effectively used as a template for creating a container
that fits into the existing LXC container framework. There is no
separate OCI runtime (e.g. containerd) being used here.

## Problem

Because all layers are currently squashed into one container disk, there
is no deduplication. When multiple containers are created from the same
base image, all base layers are always duplicated into each new
container rootfs. Furthermore, multiple OCI images might contain
identical layers. Since each OCI image bundles its layers into one TAR
archive, the layers are not deduplicated between OCI images.
Additionally, when pulling from a registry an OCI image that contains
layers already stored locally, those layers are redundantly
re-downloaded.

## Objective

Deduplicate shared layers across containers and across images via
OverlayFS. When pulling from a registry, only download layers not
already stored locally. Images are immutable, content-addressed, and
referenced by digest, so a container's rootfs can never be silently
rebased by re-pulling a tag. An administrator who deliberately wants to
rebase a container onto a newer image can still do so explicitly, by
editing the container config and pointing it at a different digest.
(This has the potential to break a container, especially if a manual
`apt upgrade` was performed in the container and then the base image is
swapped out.)

# Proposal: OverlayFS integration for Proxmox VE

An independent content type just for OCI images, with the container
rootfs remaining an ordinary "rootdir" volume on any storage,
referencing the underlying image via a new property in the container
config.

## Storage layout of new content type "oci"

Add a new content type, "oci", available only on directory-based
storages (the same kind of restriction already applied to
"vztmpl"/"backup"/"snippets").

Volume naming: `<storage>:oci/sha256-<digest>`, where the digest is the
image's own OCI ImageID (the hash of its image configuration JSON).

Directory structure of the content type:
```
oci/
+-- layers/
|   +-- sha256-111.../
|   +-- sha256-222.../
|   `-- by-blob-id/
|       +-- sha256-abc...
|       `-- sha256-def...
`-- images/
     +-- sha256-9f86d081...
     `-- sha256-2c26b46b...
```

### `layers/`

Stores the extracted contents of the immutable lower layers.
Whiteouts are stored as Linux-native whiteout nodes.
Each layer has its own directory named after its uncompressed layer
archive digest (Diff ID). These directories are strictly read-only.

`by-blob-id/` caches which blob digest corresponds to which locally
stored layer. This is needed for lookups to avoid redundant
redownloading of layers. An image manifest only supplies the digests of
the potentially compressed layer blobs. But under `layers/` the names of
the directories only hold the uncompressed digest. So this `by-blob-id/`
directory correlates compressed digests to uncompressed digests, which
lets us avoid redundant downloads of already stored layers. It contains
text files named after a layer's compressed blob digest. The content of
the file is simply the uncompressed `diff_id` string. This maps a
compressed blob hash to the local uncompressed layer directory.

Layers are never independently addressable volumes and are never
returned by volume-listing operations. They are internal bookkeeping,
comparable to how a thin-pool plugin manages internal block sharing
without exposing individual chunks as volumes.

### `images/`

Each OCI image gets its own JSON file, which is simply its corresponding
OCI image configuration. The name of the file is its SHA256 hash, which
is also the ImageID by the OCI spec. This file originates directly from
the OCI image. It holds parameters like CPU architecture, entrypoint,
environment variables... and the `rootfs.diff_ids` array specifying the
ordered stack of layers in `layers/`.

## Pulling an image from a registry

1. Fetch the image index from the registry.
2. Find & Fetch the image manifest corresponding to the correct CPU
    architecture.
3. Fetch the image configuration referenced by the manifest.
4. Check `layers/by-blob-id/<compressed-digest>` for every layer listed
    in the manifest, to determine which layers are already stored
    locally.
5. Download only the missing layers. Decompress, verify the digest, and
    extract each layer tarball into `layers/<uncompressed-digest>/`.
6. Create a `layers/by-blob-id/<compressed-digest>` text file containing
    the uncompressed digest string for every layer. Replace whiteout
    files with actual whiteout nodes.
7. Save the image configuration under `images/`.

## Container configuration & container rootfs disks

The container rootfs remains an ordinary "rootdir" volume. The only
addition is a new `ocibase` property on the "rootfs" mountpoint: a
reference to an underlying OCI image volume. This property must be
accepted only on the "rootfs" mountpoint, never on other mountpoints.
The image storage referenced by this property can be on a different
storage than the rootfs volume itself. The property must be supplied
when the "rootfs" volume has an `overlay` suffix (`vm-<vmid>-overlay`
instead of `vm-<vmid>-disk-<n>`) in its name, similar to how the
`cloudinit` suffix for VM disks (`vm-<vmid>-cloudinit` instead of
`vm-<vmid>-disk-<n>`) does not allow the `iothread` and `ro` properties.
On the other hand, the `ocibase` property must not be supplied when the
disk does not have an `overlay` suffix.

Other than the regular container rootfs disks that contain the container
file system directly, an `overlay` disk contains two directories at its
root:
```
upper/  # Writable topmost layer of the container rootfs
work/   # Empty directory needed by OverlayFS. OverlayFS needs this to 
be on the same file system as `upper/`.
```

The `overlay` suffix allows us to distinguish between an `overlay` disk
with said layout and a regular container disk with the file system
placed directly at its root.

## Creating a container

1. Allocate a new volume in the `rootdir` content type containing:
     * empty `upper/` directory
     * empty `work/` directory
2. Point `rootfs` in the container config to this disk along with the
    `ocibase` property pointing to the OCI image volume.

## Starting the container

1. If the `rootfs` in the container config points to a disk with the
    `overlay` suffix, look for the `ocibase` property to identify the
    base image.
2. On the OCI image storage, read `images/sha256-<digest>` to
    extract the `rootfs.diff_ids` array.
3. Map the `diff_ids` to their absolute paths in the `layers/` directory
    to generate the `lowerdir` string in stack order.
4. Mount the overlay filesystem.
    (Combining OverlayFS with idmapping would probably require
  https://lore.kernel.org/all/20260615-work-idmapped-overlayfs-v1-0-7381632aa402@kernel.org/
    to be applied in the kernel.)
    Very simplified example command for intuition:
    ```
    mount -t overlay overlay --map-users 0:100000:65536 --map-groups 
0:100000:65536 -o 
lowerdir=/var/lib/vz/oci/layers/sha256-111...:/var/lib/vz/oci/layers/sha256-222...,upperdir=/path/to/mountdir/of/vm-100-overlay/upper,workdir=/path/to/mountdir/of/vm-100-overlay/work 
/var/lib/lxc/100/rootfs
    ```

## Cloning a container

This requires no changes.
Since the rootfs remains an ordinary volume and the new property is an
ordinary scalar config value, existing clone/copy logic already copies
both correctly.

## Deleting an image

Because layers are deduplicated and shared, deleting a base image
requires a reference-counting mechanism:

1. Iterate through all container configs to verify that no `rootfs`
    references the image in the `ocibase` property (including pending
    changes and snapshots). Abort if any do.
2. Remove the target image configuration file in `images/`.
3. For every `diff_id` that was present in the deleted image config,
    check if any remaining config across all other stored `images/` still
    references it.
4. For any `diff_id` without references:
     * Delete its directory from `layers/`.
     * Delete all corresponding mapping files in `layers/by-blob-id/`.

## Open Questions & Notes

* Backups and migration will require special handling because container
   root filesystems are no longer fully self-contained. For migration
   probably require the image to already be present on the target. For
   backups I am unsure whether we should include the image aswell.
* Do all directory-based storage plugins support whiteout nodes?
* Human-readable image tags

---

I will follow up with patches once I have a working prototype.

Re: [RFC container/storage] OCI image OverlayFS integration proposal

[Filip Schauer]

On 14/07/2026 16:11, Jörg Behrmann wrote:
> Hi!
>
> I usually only watch from the sidelines, but this just tumbled through my mail
> client while I was waiting for a build to finish. I've not read the whole
> proposel, so excuse me if this comment is silly…
>
> On Tue, Jul 14, 2026 at 03:19:09PM +0200, Filip Schauer wrote:
>> Directory structure of the content type:
>> ```
>> oci/
>> +-- layers/
>> |   +-- sha256-111.../
>> |   +-- sha256-222.../
>> |   `-- by-blob-id/
>> |       +-- sha256-abc...
>> |       `-- sha256-def...
>> `-- images/
>>      +-- sha256-9f86d081...
>>      `-- sha256-2c26b46b...
>> ```
>>
> This looks a bit like composefs. I grepped through your proposal and didn't see
> it mentioned.
>
> - The original C project https://github.com/composefs/composefs
> - The Rust rewrite https://github.com/composefs/composefs-rs
>
> RH built this because of their bootc plans. I'm not a fan of that and also not
> much a fan of OCI containers themselves, but I know the people working on it and
> they are nice and the project does have some cool uses.
>
> There's a talk from last year's All Systems Go at [1] and you can find the
> slides at [2]. The above reminded me particularly of slide 13 and later.
>
> I hope this isn't too far off from what you want to do.
>
> best regards,
> Jörg Behrmann
>
> [1] https://media.ccc.de/v/all-systems-go-2025-362-uki-composefs-and-remote-attestation-for-bootable-containers
> [2] https://cfp.all-systems-go.io/media/all-systems-go-2025/submissions/TNKPQS/resources/2025_All_Systems_Go_-_UKI_composefs_and_r9Z95aW.pdf

Hello,

thanks for pointing this out, composefs certainly looks interesting.

As I understand it, composefs deduplicates at the individual file level,
while my proposal deduplicates at the layer level.

composefs is not yet packaged in Debian stable. That said, it might be
something to keep in mind for a potential future extension.
For the initial version however, I would rather keep things simple and
stay aligned with the layer model defined by the OCI spec.