* [RFC container/storage] OCI image OverlayFS integration proposal
@ 2026-07-14 13:19 Filip Schauer
[not found] ` <alZDjfWvqoQFKTV3@physik.fu-berlin.de>
0 siblings, 1 reply; 2+ messages in thread
From: Filip Schauer @ 2026-07-14 13:19 UTC (permalink / raw)
To: Proxmox VE development discussion
I would like to propose extending the OCI container tech preview with
OverlayFS, so that image layers are stored once (content-addressed &
deduplicated) and mounted read-only as lowerdirs, with a thin writable
upperdir per container, rather than being squashed into one rootfs
volume per container.
---
## Status Quo
In the current technology preview of OCI (Open Container Initiative)
integration, OCI images can be used as templates for LXC containers.
They can be pulled from an OCI registry into a storage that supports the
"vztmpl" (CT Templates) content type via the "Pull from OCI Registry"
button in the web UI. A downloaded OCI image is squashed into a single
TAR archive containing the config and all layers. When creating a
container, an OCI image can be selected as the "ostemplate". It is
automatically detected as an OCI template and its layers are extracted
and squashed into a single container volume (stored in the "rootdir"
content type) used as the container's rootfs. The container is also
configured according to the config supplied in the OCI image. Once the
container is created, it works just like any other LXC container. The
OCI image was effectively used as a template for creating a container
that fits into the existing LXC container framework. There is no
separate OCI runtime (e.g. containerd) being used here.
## Problem
Because all layers are currently squashed into one container disk, there
is no deduplication. When multiple containers are created from the same
base image, all base layers are always duplicated into each new
container rootfs. Furthermore, multiple OCI images might contain
identical layers. Since each OCI image bundles its layers into one TAR
archive, the layers are not deduplicated between OCI images.
Additionally, when pulling from a registry an OCI image that contains
layers already stored locally, those layers are redundantly
re-downloaded.
## Objective
Deduplicate shared layers across containers and across images via
OverlayFS. When pulling from a registry, only download layers not
already stored locally. Images are immutable, content-addressed, and
referenced by digest, so a container's rootfs can never be silently
rebased by re-pulling a tag. An administrator who deliberately wants to
rebase a container onto a newer image can still do so explicitly, by
editing the container config and pointing it at a different digest.
(This has the potential to break a container, especially if a manual
`apt upgrade` was performed in the container and then the base image is
swapped out.)
# Proposal: OverlayFS integration for Proxmox VE
An independent content type just for OCI images, with the container
rootfs remaining an ordinary "rootdir" volume on any storage,
referencing the underlying image via a new property in the container
config.
## Storage layout of new content type "oci"
Add a new content type, "oci", available only on directory-based
storages (the same kind of restriction already applied to
"vztmpl"/"backup"/"snippets").
Volume naming: `<storage>:oci/sha256-<digest>`, where the digest is the
image's own OCI ImageID (the hash of its image configuration JSON).
Directory structure of the content type:
```
oci/
+-- layers/
| +-- sha256-111.../
| +-- sha256-222.../
| `-- by-blob-id/
| +-- sha256-abc...
| `-- sha256-def...
`-- images/
+-- sha256-9f86d081...
`-- sha256-2c26b46b...
```
### `layers/`
Stores the extracted contents of the immutable lower layers.
Whiteouts are stored as Linux-native whiteout nodes.
Each layer has its own directory named after its uncompressed layer
archive digest (Diff ID). These directories are strictly read-only.
`by-blob-id/` caches which blob digest corresponds to which locally
stored layer. This is needed for lookups to avoid redundant
redownloading of layers. An image manifest only supplies the digests of
the potentially compressed layer blobs. But under `layers/` the names of
the directories only hold the uncompressed digest. So this `by-blob-id/`
directory correlates compressed digests to uncompressed digests, which
lets us avoid redundant downloads of already stored layers. It contains
text files named after a layer's compressed blob digest. The content of
the file is simply the uncompressed `diff_id` string. This maps a
compressed blob hash to the local uncompressed layer directory.
Layers are never independently addressable volumes and are never
returned by volume-listing operations. They are internal bookkeeping,
comparable to how a thin-pool plugin manages internal block sharing
without exposing individual chunks as volumes.
### `images/`
Each OCI image gets its own JSON file, which is simply its corresponding
OCI image configuration. The name of the file is its SHA256 hash, which
is also the ImageID by the OCI spec. This file originates directly from
the OCI image. It holds parameters like CPU architecture, entrypoint,
environment variables... and the `rootfs.diff_ids` array specifying the
ordered stack of layers in `layers/`.
## Pulling an image from a registry
1. Fetch the image index from the registry.
2. Find & Fetch the image manifest corresponding to the correct CPU
architecture.
3. Fetch the image configuration referenced by the manifest.
4. Check `layers/by-blob-id/<compressed-digest>` for every layer listed
in the manifest, to determine which layers are already stored
locally.
5. Download only the missing layers. Decompress, verify the digest, and
extract each layer tarball into `layers/<uncompressed-digest>/`.
6. Create a `layers/by-blob-id/<compressed-digest>` text file containing
the uncompressed digest string for every layer. Replace whiteout
files with actual whiteout nodes.
7. Save the image configuration under `images/`.
## Container configuration & container rootfs disks
The container rootfs remains an ordinary "rootdir" volume. The only
addition is a new `ocibase` property on the "rootfs" mountpoint: a
reference to an underlying OCI image volume. This property must be
accepted only on the "rootfs" mountpoint, never on other mountpoints.
The image storage referenced by this property can be on a different
storage than the rootfs volume itself. The property must be supplied
when the "rootfs" volume has an `overlay` suffix (`vm-<vmid>-overlay`
instead of `vm-<vmid>-disk-<n>`) in its name, similar to how the
`cloudinit` suffix for VM disks (`vm-<vmid>-cloudinit` instead of
`vm-<vmid>-disk-<n>`) does not allow the `iothread` and `ro` properties.
On the other hand, the `ocibase` property must not be supplied when the
disk does not have an `overlay` suffix.
Other than the regular container rootfs disks that contain the container
file system directly, an `overlay` disk contains two directories at its
root:
```
upper/ # Writable topmost layer of the container rootfs
work/ # Empty directory needed by OverlayFS. OverlayFS needs this to
be on the same file system as `upper/`.
```
The `overlay` suffix allows us to distinguish between an `overlay` disk
with said layout and a regular container disk with the file system
placed directly at its root.
## Creating a container
1. Allocate a new volume in the `rootdir` content type containing:
* empty `upper/` directory
* empty `work/` directory
2. Point `rootfs` in the container config to this disk along with the
`ocibase` property pointing to the OCI image volume.
## Starting the container
1. If the `rootfs` in the container config points to a disk with the
`overlay` suffix, look for the `ocibase` property to identify the
base image.
2. On the OCI image storage, read `images/sha256-<digest>` to
extract the `rootfs.diff_ids` array.
3. Map the `diff_ids` to their absolute paths in the `layers/` directory
to generate the `lowerdir` string in stack order.
4. Mount the overlay filesystem.
(Combining OverlayFS with idmapping would probably require
https://lore.kernel.org/all/20260615-work-idmapped-overlayfs-v1-0-7381632aa402@kernel.org/
to be applied in the kernel.)
Very simplified example command for intuition:
```
mount -t overlay overlay --map-users 0:100000:65536 --map-groups
0:100000:65536 -o
lowerdir=/var/lib/vz/oci/layers/sha256-111...:/var/lib/vz/oci/layers/sha256-222...,upperdir=/path/to/mountdir/of/vm-100-overlay/upper,workdir=/path/to/mountdir/of/vm-100-overlay/work
/var/lib/lxc/100/rootfs
```
## Cloning a container
This requires no changes.
Since the rootfs remains an ordinary volume and the new property is an
ordinary scalar config value, existing clone/copy logic already copies
both correctly.
## Deleting an image
Because layers are deduplicated and shared, deleting a base image
requires a reference-counting mechanism:
1. Iterate through all container configs to verify that no `rootfs`
references the image in the `ocibase` property (including pending
changes and snapshots). Abort if any do.
2. Remove the target image configuration file in `images/`.
3. For every `diff_id` that was present in the deleted image config,
check if any remaining config across all other stored `images/` still
references it.
4. For any `diff_id` without references:
* Delete its directory from `layers/`.
* Delete all corresponding mapping files in `layers/by-blob-id/`.
## Open Questions & Notes
* Backups and migration will require special handling because container
root filesystems are no longer fully self-contained. For migration
probably require the image to already be present on the target. For
backups I am unsure whether we should include the image aswell.
* Do all directory-based storage plugins support whiteout nodes?
* Human-readable image tags
---
I will follow up with patches once I have a working prototype.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-07-15 10:00 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-14 13:19 [RFC container/storage] OCI image OverlayFS integration proposal Filip Schauer
[not found] ` <alZDjfWvqoQFKTV3@physik.fu-berlin.de>
2026-07-15 10:00 ` Filip Schauer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox