* [PATCH proxmox-perl-rs 01/26] pve-rs: oci: update doc comment of parse_and_extract_image
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 02/26] oci: rename error when no manifest can be selected Filip Schauer
` (24 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
The function does not return the entire image config, only the optional
execution parameters contained in the image config.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
This is the same thing as 9b5bf494b175 in proxmox.git.
pve-rs/src/bindings/oci.rs | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/pve-rs/src/bindings/oci.rs b/pve-rs/src/bindings/oci.rs
index 7f5acaf..30e74fd 100644
--- a/pve-rs/src/bindings/oci.rs
+++ b/pve-rs/src/bindings/oci.rs
@@ -7,7 +7,7 @@ pub mod pve_rs_oci {
use anyhow::Error;
use proxmox_oci::Config;
- /// Extract the rootfs of an OCI image tar and return the image config.
+ /// Extract the rootfs of an OCI image tar and return its execution parameters.
///
/// # Arguments
///
@@ -15,6 +15,11 @@ pub mod pve_rs_oci {
/// * `rootfs_path` - Destination path where the rootfs will be extracted to
/// * `arch` - Optional CPU architecture used to pick the first matching manifest from a multi-arch
/// image index. If `None`, the first manifest will be used.
+ ///
+ /// # Returns
+ ///
+ /// On success, this returns the optional container execution parameters
+ /// if present in the image configuration.
#[export]
pub fn parse_and_extract_image(
oci_tar_path: &str,
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 02/26] oci: rename error when no manifest can be selected
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 01/26] pve-rs: oci: update doc comment of parse_and_extract_image Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 03/26] oci: add `os` parameter to parse_and_extract_image Filip Schauer
` (23 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Selecting an image manifest from the image index can fail for reasons
other than the CPU architecture.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/lib.rs | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index 78045f49..0e732324 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -136,8 +136,8 @@ pub enum ParseError {
WrongMediaType,
#[error("IO error: {0}")]
Io(#[from] io::Error),
- #[error("Unsupported CPU architecture")]
- UnsupportedArchitecture,
+ #[error("No image manifest found in image index for the requested platform")]
+ NoMatchingManifest,
#[error("Missing image config")]
MissingImageConfig,
}
@@ -151,7 +151,7 @@ fn parse_image<P: AsRef<Path>>(
let image_manifest = oci_tar_image
.image_manifest(arch)
- .ok_or(ParseError::UnsupportedArchitecture)??;
+ .ok_or(ParseError::NoMatchingManifest)??;
let image_config_descriptor = image_manifest.config();
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 03/26] oci: add `os` parameter to parse_and_extract_image
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 01/26] pve-rs: oci: update doc comment of parse_and_extract_image Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 02/26] oci: rename error when no manifest can be selected Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 04/26] pve-rs: " Filip Schauer
` (22 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Manifest selection from a multi-platform image index only compared the
architecture. Add an optional `os` argument to `parse_and_extract_image`
and match it together with `arch`.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/lib.rs | 10 +++++++---
proxmox-oci/src/oci_tar_image.rs | 9 ++++++---
proxmox-oci/tests/helper/mod.rs | 9 +++++++--
3 files changed, 20 insertions(+), 8 deletions(-)
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index 0e732324..f2664857 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -8,7 +8,7 @@ use std::str::FromStr;
use flate2::read::GzDecoder;
use oci_spec::OciSpecError;
-pub use oci_spec::image::{Arch, Config};
+pub use oci_spec::image::{Arch, Config, Os};
use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType};
use sha2::digest::generic_array::GenericArray;
use sha2::{Digest, Sha256};
@@ -109,6 +109,8 @@ pub enum ProxmoxOciError {
///
/// * `oci_tar_path` - Path to the OCI image tar archive
/// * `rootfs_path` - Destination path where the rootfs will be extracted to
+/// * `os` - Optional operating system used to pick the first matching manifest from a multi-arch
+/// image index. If `None`, the first manifest will be used.
/// * `arch` - Optional CPU architecture used to pick the first matching manifest from a multi-arch
/// image index. If `None`, the first manifest will be used.
///
@@ -119,9 +121,10 @@ pub enum ProxmoxOciError {
pub fn parse_and_extract_image<P: AsRef<Path>>(
oci_tar_path: P,
rootfs_path: P,
+ os: Option<&Os>,
arch: Option<&Arch>,
) -> Result<Option<Config>, ProxmoxOciError> {
- let (oci_tar_image, image_manifest, image_config) = parse_image(oci_tar_path, arch)?;
+ let (oci_tar_image, image_manifest, image_config) = parse_image(oci_tar_path, os, arch)?;
extract_image_rootfs(oci_tar_image, &image_manifest, &image_config, rootfs_path)?;
@@ -144,13 +147,14 @@ pub enum ParseError {
fn parse_image<P: AsRef<Path>>(
oci_tar_path: P,
+ os: Option<&Os>,
arch: Option<&Arch>,
) -> Result<(OciTarImage<File>, ImageManifest, ImageConfiguration), ParseError> {
let oci_tar_file = File::open(oci_tar_path)?;
let mut oci_tar_image = OciTarImage::new(oci_tar_file)?;
let image_manifest = oci_tar_image
- .image_manifest(arch)
+ .image_manifest(os, arch)
.ok_or(ParseError::NoMatchingManifest)??;
let image_config_descriptor = image_manifest.config();
diff --git a/proxmox-oci/src/oci_tar_image.rs b/proxmox-oci/src/oci_tar_image.rs
index ebe3edb4..f51556e9 100644
--- a/proxmox-oci/src/oci_tar_image.rs
+++ b/proxmox-oci/src/oci_tar_image.rs
@@ -4,7 +4,7 @@ use std::ops::Range;
use std::path::{Path, PathBuf};
use oci_spec::OciSpecError;
-use oci_spec::image::{Arch, Digest, ImageIndex, ImageManifest, MediaType};
+use oci_spec::image::{Arch, Digest, ImageIndex, ImageManifest, MediaType, Os};
use tar::Archive;
use proxmox_io::RangeReader;
@@ -73,12 +73,15 @@ impl<R: Read + Seek> OciTarImage<R> {
pub fn image_manifest(
&mut self,
+ os: Option<&Os>,
architecture: Option<&Arch>,
) -> Option<oci_spec::Result<ImageManifest>> {
let digest = match self.image_index.manifests().iter().find(|d| {
d.media_type() == &MediaType::ImageManifest
- && architecture
- .is_none_or(|a| d.platform().as_ref().is_none_or(|p| p.architecture() == a))
+ && d.platform().as_ref().is_none_or(|p| {
+ architecture.is_none_or(|a| p.architecture() == a)
+ && os.is_none_or(|os| p.os() == os)
+ })
}) {
Some(descriptor) => descriptor.digest(),
None => return None,
diff --git a/proxmox-oci/tests/helper/mod.rs b/proxmox-oci/tests/helper/mod.rs
index c07bbf9c..46491a89 100644
--- a/proxmox-oci/tests/helper/mod.rs
+++ b/proxmox-oci/tests/helper/mod.rs
@@ -5,7 +5,7 @@ use std::path::{Path, PathBuf};
use anyhow::Result;
-use proxmox_oci::{Arch, Config, ProxmoxOciError, parse_and_extract_image};
+use proxmox_oci::{Arch, Config, Os, ProxmoxOciError, parse_and_extract_image};
use proxmox_sys::fs::make_tmp_dir;
mod oci_tar_image_builder;
@@ -71,7 +71,12 @@ impl TestImage {
}
pub fn extract(&self) -> Result<Option<Config>, ProxmoxOciError> {
- parse_and_extract_image(&self.path, &self.extract_dir, Some(&Arch::Amd64))
+ parse_and_extract_image(
+ &self.path,
+ &self.extract_dir,
+ Some(&Os::Linux),
+ Some(&Arch::Amd64),
+ )
}
}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox-perl-rs 04/26] pve-rs: oci: add `os` parameter to parse_and_extract_image
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (2 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 03/26] oci: add `os` parameter to parse_and_extract_image Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 05/26] oci: add a digest helper module Filip Schauer
` (21 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
pve-rs/src/bindings/oci.rs | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/pve-rs/src/bindings/oci.rs b/pve-rs/src/bindings/oci.rs
index 30e74fd..11ac07f 100644
--- a/pve-rs/src/bindings/oci.rs
+++ b/pve-rs/src/bindings/oci.rs
@@ -13,6 +13,8 @@ pub mod pve_rs_oci {
///
/// * `oci_tar_path` - Path to the OCI image tar archive
/// * `rootfs_path` - Destination path where the rootfs will be extracted to
+ /// * `os` - Optional operating system used to pick the first matching manifest from a multi-arch
+ /// image index. If `None`, the first manifest will be used.
/// * `arch` - Optional CPU architecture used to pick the first matching manifest from a multi-arch
/// image index. If `None`, the first manifest will be used.
///
@@ -24,11 +26,17 @@ pub mod pve_rs_oci {
pub fn parse_and_extract_image(
oci_tar_path: &str,
rootfs_path: &str,
+ os: Option<&str>,
arch: Option<&str>,
) -> Result<Config, Error> {
let arch = arch.map(Into::into);
- proxmox_oci::parse_and_extract_image(oci_tar_path, rootfs_path, arch.as_ref())
- .map(|config| config.unwrap_or_default())
- .map_err(Into::into)
+ proxmox_oci::parse_and_extract_image(
+ oci_tar_path,
+ rootfs_path,
+ os.map(Into::into).as_ref(),
+ arch.as_ref(),
+ )
+ .map(|config| config.unwrap_or_default())
+ .map_err(Into::into)
}
}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 05/26] oci: add a digest helper module
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (3 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 04/26] pve-rs: " Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 06/26] oci: reduce redundant layer decompression during extraction Filip Schauer
` (20 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/digest.rs | 94 +++++++++++++++++++++++++++++++++++++++
proxmox-oci/src/lib.rs | 2 +
2 files changed, 96 insertions(+)
create mode 100644 proxmox-oci/src/digest.rs
diff --git a/proxmox-oci/src/digest.rs b/proxmox-oci/src/digest.rs
new file mode 100644
index 00000000..b2170b87
--- /dev/null
+++ b/proxmox-oci/src/digest.rs
@@ -0,0 +1,94 @@
+use std::io::{self, Read};
+use std::str::FromStr;
+
+use oci_spec::image::{Digest, DigestAlgorithm};
+use sha2::Digest as _;
+use sha2::digest::DynDigest;
+use thiserror::Error;
+
+#[derive(Debug, Error)]
+pub enum DigestError {
+ #[error("Unsupported digest algorithm: {0}")]
+ UnsupportedAlgorithm(String),
+ #[error("Digest mismatch. Expected {expected}, got {actual}")]
+ Mismatch { expected: Digest, actual: Digest },
+}
+
+struct Digester(Box<dyn DynDigest>);
+
+impl Digester {
+ fn new(digest_algorithm: &DigestAlgorithm) -> Result<Self, DigestError> {
+ let hasher: Box<dyn DynDigest> = match digest_algorithm {
+ DigestAlgorithm::Sha256 => Box::new(sha2::Sha256::new()),
+ DigestAlgorithm::Sha384 => Box::new(sha2::Sha384::new()),
+ DigestAlgorithm::Sha512 => Box::new(sha2::Sha512::new()),
+ other => return Err(DigestError::UnsupportedAlgorithm(other.to_string())),
+ };
+ Ok(Self(hasher))
+ }
+
+ fn update(&mut self, data: &[u8]) {
+ self.0.update(data);
+ }
+
+ fn finalize(&mut self) -> Digest {
+ let sha_bit_count = self.0.output_size() * 8;
+ let digest_str = self
+ .0
+ .finalize_reset()
+ .iter()
+ .map(|b| format!("{b:02x}"))
+ .collect::<String>();
+ Digest::from_str(&format!("sha{sha_bit_count}:{digest_str}")).expect("valid digest")
+ }
+}
+
+pub struct VerifyingReader<R: Read> {
+ inner: R,
+ digester: Digester,
+ expected_digest: Digest,
+ eof: bool,
+}
+
+impl<R: Read> VerifyingReader<R> {
+ pub fn new(inner: R, expected_digest: Digest) -> Result<Self, DigestError> {
+ Ok(Self {
+ inner,
+ digester: Digester::new(expected_digest.algorithm())?,
+ expected_digest,
+ eof: false,
+ })
+ }
+
+ fn verify(&mut self) -> io::Result<()> {
+ let actual = self.digester.finalize();
+
+ if actual != self.expected_digest {
+ return Err(io::Error::other(DigestError::Mismatch {
+ expected: self.expected_digest.clone(),
+ actual,
+ }));
+ }
+
+ Ok(())
+ }
+}
+
+impl<R: Read> Read for VerifyingReader<R> {
+ fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
+ if buf.is_empty() {
+ return Ok(0);
+ }
+
+ let n = self.inner.read(buf)?;
+ if n == 0 {
+ if !self.eof {
+ self.eof = true;
+ self.verify()?;
+ }
+ return Ok(0);
+ }
+ self.digester.update(&buf[..n]);
+ Ok(n)
+ }
+}
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index f2664857..f6dd2e51 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -15,6 +15,8 @@ use sha2::{Digest, Sha256};
use tar::{Archive, EntryType};
use thiserror::Error;
+mod digest;
+
mod oci_tar_image;
use oci_tar_image::{OciTarImage, OciTarImageBlob};
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 06/26] oci: reduce redundant layer decompression during extraction
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (4 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 05/26] oci: add a digest helper module Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 07/26] oci: factor out parsing and extraction into modules Filip Schauer
` (19 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
According to the OCI Image Specification, the order of layers in the
image manifest matches the order of the rootfs.diff_ids in the image
configuration.
Instead of redundantly decompressing and hashing every layer upfront to
build a lookup map, pair layer blobs directly with their corresponding
diff_ids and verify the digests on the fly.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/lib.rs | 115 +++++++++--------------------------------
1 file changed, 24 insertions(+), 91 deletions(-)
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index f6dd2e51..fd1f004b 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -1,4 +1,3 @@
-use std::collections::HashMap;
use std::ffi::OsStr;
use std::fs::{File, read_dir, remove_dir_all, remove_file};
use std::io::{self, ErrorKind, Read, Seek, SeekFrom};
@@ -10,12 +9,11 @@ use flate2::read::GzDecoder;
use oci_spec::OciSpecError;
pub use oci_spec::image::{Arch, Config, Os};
use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType};
-use sha2::digest::generic_array::GenericArray;
-use sha2::{Digest, Sha256};
use tar::{Archive, EntryType};
use thiserror::Error;
mod digest;
+use digest::{DigestError, VerifyingReader};
mod oci_tar_image;
use oci_tar_image::{OciTarImage, OciTarImageBlob};
@@ -23,80 +21,6 @@ use oci_tar_image::{OciTarImage, OciTarImageBlob};
const WHITEOUT_PREFIX: &[u8] = b".wh.";
const OPAQUE_WHITEOUT_NAME: &str = ".wh..wh..opq";
-fn compute_digest<R: Read, H: Digest>(
- mut reader: R,
- mut hasher: H,
-) -> io::Result<GenericArray<u8, H::OutputSize>> {
- let mut buf = proxmox_io::boxed::zeroed(32768);
-
- loop {
- let bytes_read = reader.read(&mut buf)?;
- if bytes_read == 0 {
- break Ok(hasher.finalize());
- }
-
- hasher.update(&buf[..bytes_read]);
- }
-}
-
-fn compute_sha256<R: Read>(reader: R) -> io::Result<oci_spec::image::Sha256Digest> {
- let digest = compute_digest(reader, Sha256::new())?;
- Ok(oci_spec::image::Sha256Digest::from_str(&format!("{digest:x}")).expect("valid digest"))
-}
-
-/// Build a mapping from uncompressed layer digests (as found in the image config's `rootfs.diff_ids`)
-/// to their corresponding compressed-layer digests (i.e. the filenames under `blobs/<algorithm>/<digest>`)
-fn build_layer_map<R: Read + Seek>(
- mut oci_tar_image: OciTarImage<R>,
- image_manifest: &ImageManifest,
-) -> Result<
- (
- OciTarImage<R>,
- HashMap<oci_spec::image::Digest, oci_spec::image::Descriptor>,
- ),
- ExtractError,
-> {
- let mut layer_mapping = HashMap::new();
-
- for layer in image_manifest.layers() {
- let digest = match layer.media_type() {
- MediaType::ImageLayer | MediaType::ImageLayerNonDistributable => layer.digest().clone(),
- MediaType::ImageLayerGzip | MediaType::ImageLayerNonDistributableGzip => {
- let mut compressed_blob = oci_tar_image
- .open_blob(layer.digest())
- .ok_or(ExtractError::MissingLayerFile(layer.digest().clone()))?;
- let decoder = GzDecoder::new(&mut compressed_blob);
- let hash = compute_sha256(decoder)?.into();
- oci_tar_image = compressed_blob.into_oci_tar_image();
- hash
- }
- MediaType::ImageLayerZstd | MediaType::ImageLayerNonDistributableZstd => {
- let mut compressed_blob = oci_tar_image
- .open_blob(layer.digest())
- .ok_or(ExtractError::MissingLayerFile(layer.digest().clone()))?;
- let decoder = zstd::Decoder::new(&mut compressed_blob)?;
- let hash = compute_sha256(decoder)?.into();
- oci_tar_image = compressed_blob.into_oci_tar_image();
- hash
- }
- // Skip any other non-ImageLayer related media types.
- // Match explicitly to avoid missing new image layer types when oci-spec updates.
- MediaType::Descriptor
- | MediaType::LayoutHeader
- | MediaType::ImageManifest
- | MediaType::ImageIndex
- | MediaType::ImageConfig
- | MediaType::ArtifactManifest
- | MediaType::EmptyJSON
- | MediaType::Other(_) => continue,
- };
-
- layer_mapping.insert(digest, layer.clone());
- }
-
- Ok((oci_tar_image, layer_mapping))
-}
-
#[derive(Debug, Error)]
pub enum ProxmoxOciError {
#[error("Error while parsing OCI image: {0}")]
@@ -181,8 +105,8 @@ fn parse_image<P: AsRef<Path>>(
pub enum ExtractError {
#[error("Incorrectly formatted digest: \"{0}\"")]
InvalidDigest(String),
- #[error("Unknown layer digest {0} found in rootfs.diff_ids")]
- UnknownLayerDigest(oci_spec::image::Digest),
+ #[error(transparent)]
+ DigestError(#[from] DigestError),
#[error("Layer file {0} mentioned in image manifest is missing")]
MissingLayerFile(oci_spec::image::Digest),
#[error("IO error: {0}")]
@@ -192,23 +116,23 @@ pub enum ExtractError {
}
fn extract_image_rootfs<R: Read + Seek, P: AsRef<Path>>(
- oci_tar_image: OciTarImage<R>,
+ mut oci_tar_image: OciTarImage<R>,
image_manifest: &ImageManifest,
image_config: &ImageConfiguration,
target_path: P,
) -> Result<(), ExtractError> {
- let (mut oci_tar_image, layer_map) = build_layer_map(oci_tar_image, image_manifest)?;
let target_path = target_path.as_ref().canonicalize()?;
- for layer in image_config.rootfs().diff_ids() {
- let layer_digest = oci_spec::image::Digest::from_str(layer)
- .map_err(|_| ExtractError::InvalidDigest(layer.to_string()))?;
- let layer_descriptor = layer_map
- .get(&layer_digest)
- .ok_or(ExtractError::UnknownLayerDigest(layer_digest.clone()))?;
- let mut layer_file = oci_tar_image
- .open_blob(layer_descriptor.digest())
- .ok_or(ExtractError::MissingLayerFile(layer_digest))?;
+ for (layer_descriptor, diffid_str) in image_manifest
+ .layers()
+ .iter()
+ .zip(image_config.rootfs().diff_ids())
+ {
+ let layer_diffid = oci_spec::image::Digest::from_str(diffid_str)
+ .map_err(|_| ExtractError::InvalidDigest(diffid_str.to_string()))?;
+ let mut layer_file = oci_tar_image.open_blob(layer_descriptor.digest()).ok_or(
+ ExtractError::MissingLayerFile(layer_descriptor.digest().clone()),
+ )?;
type DecodeFn<T> = Box<dyn for<'a> Fn(&'a mut T) -> io::Result<Box<dyn Read + 'a>>>;
let decode_fn: DecodeFn<OciTarImageBlob<R>> = match layer_descriptor.media_type() {
@@ -235,7 +159,11 @@ fn extract_image_rootfs<R: Read + Seek, P: AsRef<Path>>(
}
};
- remove_replaced_paths(&mut decode_fn(&mut layer_file)?, &target_path)?;
+ remove_replaced_paths(
+ &mut VerifyingReader::new(decode_fn(&mut layer_file)?, layer_diffid)?,
+ &target_path,
+ )?;
+
layer_file.seek(SeekFrom::Start(0))?;
extract_archive(&mut decode_fn(&mut layer_file)?, &target_path)?;
@@ -297,6 +225,8 @@ fn resolve_dir_in_target(target_path: &Path, dir: &Path) -> io::Result<Option<Pa
}
/// Remove what a layer replaces with directories or whites out from previous layers.
+///
+/// Reads `reader` to EOF before removing anything.
fn remove_replaced_paths<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()> {
let mut archive = Archive::new(reader);
@@ -314,6 +244,9 @@ fn remove_replaced_paths<R: Read>(reader: &mut R, target_path: &Path) -> io::Res
}
}
+ // Force an EOF so VerifyingReader actually verifies.
+ io::copy(archive.into_inner(), &mut io::sink())?;
+
// Parents first and before whiteouts, so nothing resolves through a symlink being replaced.
directories.sort();
for dir in directories {
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 07/26] oci: factor out parsing and extraction into modules
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (5 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 06/26] oci: reduce redundant layer decompression during extraction Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 08/26] oci: explicitly set mtime of directories when extracting layer Filip Schauer
` (18 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
| 273 +++++++++++++++++++++++++++++++
proxmox-oci/src/lib.rs | 324 ++-----------------------------------
proxmox-oci/src/parse.rs | 52 ++++++
3 files changed, 334 insertions(+), 315 deletions(-)
create mode 100644 proxmox-oci/src/extract.rs
create mode 100644 proxmox-oci/src/parse.rs
--git a/proxmox-oci/src/extract.rs b/proxmox-oci/src/extract.rs
new file mode 100644
index 00000000..d9d5f1b4
--- /dev/null
+++ b/proxmox-oci/src/extract.rs
@@ -0,0 +1,273 @@
+use std::ffi::OsStr;
+use std::fs::{read_dir, remove_dir_all, remove_file};
+use std::io::{self, ErrorKind, Read, Seek, SeekFrom};
+use std::os::unix::ffi::OsStrExt;
+use std::path::{Path, PathBuf};
+use std::str::FromStr;
+
+use flate2::read::GzDecoder;
+use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType};
+use tar::{Archive, EntryType};
+use thiserror::Error;
+
+use crate::digest::{DigestError, VerifyingReader};
+use crate::oci_tar_image::{OciTarImage, OciTarImageBlob};
+
+const WHITEOUT_PREFIX: &[u8] = b".wh.";
+const OPAQUE_WHITEOUT_NAME: &str = ".wh..wh..opq";
+
+#[derive(Debug, Error)]
+pub enum ExtractError {
+ #[error("Incorrectly formatted digest: \"{0}\"")]
+ InvalidDigest(String),
+ #[error(transparent)]
+ DigestError(#[from] DigestError),
+ #[error("Layer file {0} mentioned in image manifest is missing")]
+ MissingLayerFile(oci_spec::image::Digest),
+ #[error(transparent)]
+ Io(#[from] io::Error),
+ #[error("Layer has wrong media type: {0}")]
+ WrongMediaType(String),
+}
+
+pub(crate) fn extract_image_rootfs<R: Read + Seek, P: AsRef<Path>>(
+ mut oci_tar_image: OciTarImage<R>,
+ image_manifest: &ImageManifest,
+ image_config: &ImageConfiguration,
+ target_path: P,
+) -> Result<(), ExtractError> {
+ let target_path = target_path.as_ref().canonicalize()?;
+
+ for (layer_descriptor, diffid_str) in image_manifest
+ .layers()
+ .iter()
+ .zip(image_config.rootfs().diff_ids())
+ {
+ let layer_diffid = oci_spec::image::Digest::from_str(diffid_str)
+ .map_err(|_| ExtractError::InvalidDigest(diffid_str.to_string()))?;
+ let mut layer_file = oci_tar_image.open_blob(layer_descriptor.digest()).ok_or(
+ ExtractError::MissingLayerFile(layer_descriptor.digest().clone()),
+ )?;
+
+ type DecodeFn<T> = Box<dyn for<'a> Fn(&'a mut T) -> io::Result<Box<dyn Read + 'a>>>;
+ let decode_fn: DecodeFn<OciTarImageBlob<R>> = match layer_descriptor.media_type() {
+ MediaType::ImageLayer | MediaType::ImageLayerNonDistributable => {
+ Box::new(|file| Ok(Box::new(file)))
+ }
+ MediaType::ImageLayerGzip | MediaType::ImageLayerNonDistributableGzip => {
+ Box::new(|file| Ok(Box::new(GzDecoder::new(file))))
+ }
+ MediaType::ImageLayerZstd | MediaType::ImageLayerNonDistributableZstd => {
+ Box::new(|file| Ok(Box::new(zstd::Decoder::new(file)?)))
+ }
+ // Error on any other non-ImageLayer related media types.
+ // Match explicitly to avoid missing new image layer types when oci-spec updates.
+ media_type @ (MediaType::Descriptor
+ | MediaType::LayoutHeader
+ | MediaType::ImageManifest
+ | MediaType::ImageIndex
+ | MediaType::ImageConfig
+ | MediaType::ArtifactManifest
+ | MediaType::EmptyJSON
+ | MediaType::Other(_)) => {
+ return Err(ExtractError::WrongMediaType(media_type.to_string()));
+ }
+ };
+
+ remove_replaced_paths(
+ &mut VerifyingReader::new(decode_fn(&mut layer_file)?, layer_diffid)?,
+ &target_path,
+ )?;
+
+ layer_file.seek(SeekFrom::Start(0))?;
+ extract_archive(&mut decode_fn(&mut layer_file)?, &target_path)?;
+
+ oci_tar_image = layer_file.into_oci_tar_image();
+ }
+
+ Ok(())
+}
+
+/// Validate a path previously received from a `tar::Entry`.
+///
+/// Tar files may contain arbitrary paths, so reject absolute paths and paths containing parent
+/// directory references, as joining them onto the target directory would lead outside of it.
+pub(crate) fn validate_path(path: &Path) -> io::Result<()> {
+ use std::path::Component;
+
+ if !path
+ .components()
+ .all(|c| matches!(c, Component::Normal(_) | Component::CurDir))
+ {
+ return Err(io::Error::other(format!(
+ "illegal path {path:?} in layer - absolute paths and '..' components are not allowed"
+ )));
+ }
+
+ Ok(())
+}
+
+/// Join the `path` of a layer entry onto the resolved `target_path`, resolving its parent only.
+///
+/// Fails if the parent leads outside of the target through a symlink, like the tar crate checks
+/// before unpacking, assuming that nothing else modifies the target meanwhile. Returns `None` if
+/// the parent does not exist or `path` has no last component.
+fn resolve_in_target(target_path: &Path, path: &Path) -> io::Result<Option<PathBuf>> {
+ let (Some(parent), Some(name)) = (path.parent(), path.file_name()) else {
+ return Ok(None);
+ };
+ Ok(resolve_dir_in_target(target_path, parent)?.map(|dir| dir.join(name)))
+}
+
+/// Resolve the directory `dir` of a layer entry within the resolved `target_path`, see
+/// [`resolve_in_target`].
+fn resolve_dir_in_target(target_path: &Path, dir: &Path) -> io::Result<Option<PathBuf>> {
+ let resolved = match target_path.join(dir).canonicalize() {
+ Ok(resolved) => resolved,
+ Err(err) if matches!(err.kind(), ErrorKind::NotFound | ErrorKind::NotADirectory) => {
+ return Ok(None);
+ }
+ Err(err) => return Err(err),
+ };
+
+ if !resolved.starts_with(target_path) {
+ return Err(io::Error::other(format!(
+ "illegal path {dir:?} in layer - a symlink leads outside of the target directory"
+ )));
+ }
+
+ Ok(Some(resolved))
+}
+
+/// Remove what a layer replaces with directories or whites out from previous layers.
+///
+/// Reads `reader` to EOF before removing anything.
+fn remove_replaced_paths<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()> {
+ let mut archive = Archive::new(reader);
+
+ let mut directories = Vec::new();
+ let mut whiteouts = Vec::new();
+ for entry in archive.entries()? {
+ let entry = entry?;
+ let path = entry.path()?;
+ validate_path(&path)?;
+
+ match entry.header().entry_type() {
+ EntryType::Directory => directories.push(path.into_owned()),
+ EntryType::Regular if is_whiteout(&path) => whiteouts.push(path.into_owned()),
+ _ => {}
+ }
+ }
+
+ // Force an EOF so VerifyingReader actually verifies.
+ io::copy(archive.into_inner(), &mut io::sink())?;
+
+ // Parents first and before whiteouts, so nothing resolves through a symlink being replaced.
+ directories.sort();
+ for dir in directories {
+ if let Some(dir_path_abs) = resolve_in_target(target_path, &dir)?
+ && dir_path_abs.symlink_metadata().is_ok_and(|m| !m.is_dir())
+ {
+ remove_file(dir_path_abs)?;
+ }
+ }
+
+ for whiteout in whiteouts {
+ apply_whiteout(target_path, &whiteout)?;
+ }
+
+ Ok(())
+}
+
+fn is_whiteout(path: &Path) -> bool {
+ path.file_name()
+ .is_some_and(|name| name.as_bytes().starts_with(WHITEOUT_PREFIX))
+}
+
+/// Apply the whiteout file at `path` on previous layers.
+fn apply_whiteout(target_path: &Path, path: &Path) -> io::Result<()> {
+ let Some(filename) = path.file_name() else {
+ return Ok(());
+ };
+
+ if filename == OPAQUE_WHITEOUT_NAME {
+ if let Some(parent) = path.parent()
+ && let Some(whiteout_abs_path) = resolve_dir_in_target(target_path, parent)?
+ {
+ for direntry in read_dir(whiteout_abs_path)? {
+ remove_path(&direntry?.path())?;
+ }
+ }
+ } else if let Some(filename_stripped) = filename.as_bytes().strip_prefix(WHITEOUT_PREFIX) {
+ let whiteout_path = match filename_stripped {
+ b"." => match path.parent() {
+ Some(p) if p.parent().is_some() => p,
+ _ => return Ok(()), // Prevent whiteout of root directory
+ },
+ b".." => return Ok(()), // Prevent whiteout of grandparent directory
+ fname => &path.with_file_name(OsStr::from_bytes(fname)),
+ };
+ if let Some(whiteout_abs_path) = resolve_in_target(target_path, whiteout_path)? {
+ remove_path(&whiteout_abs_path)?;
+ }
+ }
+
+ Ok(())
+}
+
+fn extract_archive<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()> {
+ let mut archive = Archive::new(reader);
+ archive.set_preserve_ownerships(true);
+ archive.set_preserve_permissions(true);
+ archive.set_unpack_xattrs(true);
+
+ // Delay directory entries until the end (they will be created if needed by descendants),
+ // to ensure that directory permissions do not interfere with descendant extraction.
+ let mut directories = Vec::new();
+ for entry in archive.entries()? {
+ let mut file = entry?;
+
+ let filepath = file.path()?;
+ validate_path(&filepath)?;
+
+ if file.header().entry_type() == EntryType::Directory {
+ directories.push(file);
+ continue;
+ } else if file.header().entry_type() == EntryType::Regular && is_whiteout(&filepath) {
+ continue;
+ }
+
+ // If a file or directory already exists at this path, remove it first.
+ if let Some(file_path_abs) = resolve_in_target(target_path, &filepath)? {
+ remove_path(&file_path_abs)?;
+ }
+
+ file.unpack_in(target_path)?;
+ }
+
+ // Apply the directories in reverse topological order,
+ // to avoid failure on restrictive parent directory permissions.
+ directories.sort_by(|a, b| b.path_bytes().cmp(&a.path_bytes()));
+ for mut dir in directories {
+ // Skip a directory replaced by a later entry, also to not apply it through a symlink.
+ if let Some(dir_path_abs) = resolve_in_target(target_path, &dir.path()?)?
+ && dir_path_abs.symlink_metadata().is_ok_and(|m| !m.is_dir())
+ {
+ continue;
+ }
+
+ dir.unpack_in(target_path)?;
+ }
+
+ Ok(())
+}
+
+/// Remove whatever exists at `path`, without following a symlink there.
+fn remove_path(path: &Path) -> io::Result<()> {
+ match path.symlink_metadata() {
+ Ok(metadata) if metadata.is_dir() => remove_dir_all(path),
+ Ok(_) => remove_file(path),
+ Err(err) if matches!(err.kind(), ErrorKind::NotFound | ErrorKind::NotADirectory) => Ok(()),
+ Err(err) => Err(err),
+ }
+}
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index fd1f004b..fcb81b03 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -1,25 +1,19 @@
-use std::ffi::OsStr;
-use std::fs::{File, read_dir, remove_dir_all, remove_file};
-use std::io::{self, ErrorKind, Read, Seek, SeekFrom};
-use std::os::unix::ffi::OsStrExt;
-use std::path::{Path, PathBuf};
-use std::str::FromStr;
+use std::path::Path;
-use flate2::read::GzDecoder;
-use oci_spec::OciSpecError;
pub use oci_spec::image::{Arch, Config, Os};
-use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType};
-use tar::{Archive, EntryType};
use thiserror::Error;
mod digest;
-use digest::{DigestError, VerifyingReader};
-mod oci_tar_image;
-use oci_tar_image::{OciTarImage, OciTarImageBlob};
+mod extract;
+pub use extract::ExtractError;
+use extract::extract_image_rootfs;
+
+mod parse;
+pub use parse::ParseError;
+use parse::parse_image;
-const WHITEOUT_PREFIX: &[u8] = b".wh.";
-const OPAQUE_WHITEOUT_NAME: &str = ".wh..wh..opq";
+mod oci_tar_image;
#[derive(Debug, Error)]
pub enum ProxmoxOciError {
@@ -56,303 +50,3 @@ pub fn parse_and_extract_image<P: AsRef<Path>>(
Ok(image_config.config().clone())
}
-
-#[derive(Debug, Error)]
-pub enum ParseError {
- #[error("OCI spec error: {0}")]
- OciSpec(#[from] OciSpecError),
- #[error("Wrong media type")]
- WrongMediaType,
- #[error("IO error: {0}")]
- Io(#[from] io::Error),
- #[error("No image manifest found in image index for the requested platform")]
- NoMatchingManifest,
- #[error("Missing image config")]
- MissingImageConfig,
-}
-
-fn parse_image<P: AsRef<Path>>(
- oci_tar_path: P,
- os: Option<&Os>,
- arch: Option<&Arch>,
-) -> Result<(OciTarImage<File>, ImageManifest, ImageConfiguration), ParseError> {
- let oci_tar_file = File::open(oci_tar_path)?;
- let mut oci_tar_image = OciTarImage::new(oci_tar_file)?;
-
- let image_manifest = oci_tar_image
- .image_manifest(os, arch)
- .ok_or(ParseError::NoMatchingManifest)??;
-
- let image_config_descriptor = image_manifest.config();
-
- if image_config_descriptor.media_type() != &MediaType::ImageConfig {
- return Err(ParseError::WrongMediaType);
- }
-
- let mut image_config_file = oci_tar_image
- .open_blob(image_config_descriptor.digest())
- .ok_or(ParseError::MissingImageConfig)?;
- let image_config = ImageConfiguration::from_reader(&mut image_config_file)?;
-
- Ok((
- image_config_file.into_oci_tar_image(),
- image_manifest,
- image_config,
- ))
-}
-
-#[derive(Debug, Error)]
-pub enum ExtractError {
- #[error("Incorrectly formatted digest: \"{0}\"")]
- InvalidDigest(String),
- #[error(transparent)]
- DigestError(#[from] DigestError),
- #[error("Layer file {0} mentioned in image manifest is missing")]
- MissingLayerFile(oci_spec::image::Digest),
- #[error("IO error: {0}")]
- Io(#[from] io::Error),
- #[error("Layer has wrong media type: {0}")]
- WrongMediaType(String),
-}
-
-fn extract_image_rootfs<R: Read + Seek, P: AsRef<Path>>(
- mut oci_tar_image: OciTarImage<R>,
- image_manifest: &ImageManifest,
- image_config: &ImageConfiguration,
- target_path: P,
-) -> Result<(), ExtractError> {
- let target_path = target_path.as_ref().canonicalize()?;
-
- for (layer_descriptor, diffid_str) in image_manifest
- .layers()
- .iter()
- .zip(image_config.rootfs().diff_ids())
- {
- let layer_diffid = oci_spec::image::Digest::from_str(diffid_str)
- .map_err(|_| ExtractError::InvalidDigest(diffid_str.to_string()))?;
- let mut layer_file = oci_tar_image.open_blob(layer_descriptor.digest()).ok_or(
- ExtractError::MissingLayerFile(layer_descriptor.digest().clone()),
- )?;
-
- type DecodeFn<T> = Box<dyn for<'a> Fn(&'a mut T) -> io::Result<Box<dyn Read + 'a>>>;
- let decode_fn: DecodeFn<OciTarImageBlob<R>> = match layer_descriptor.media_type() {
- MediaType::ImageLayer | MediaType::ImageLayerNonDistributable => {
- Box::new(|file| Ok(Box::new(file)))
- }
- MediaType::ImageLayerGzip | MediaType::ImageLayerNonDistributableGzip => {
- Box::new(|file| Ok(Box::new(GzDecoder::new(file))))
- }
- MediaType::ImageLayerZstd | MediaType::ImageLayerNonDistributableZstd => {
- Box::new(|file| Ok(Box::new(zstd::Decoder::new(file)?)))
- }
- // Error on any other non-ImageLayer related media types.
- // Match explicitly to avoid missing new image layer types when oci-spec updates.
- media_type @ (MediaType::Descriptor
- | MediaType::LayoutHeader
- | MediaType::ImageManifest
- | MediaType::ImageIndex
- | MediaType::ImageConfig
- | MediaType::ArtifactManifest
- | MediaType::EmptyJSON
- | MediaType::Other(_)) => {
- return Err(ExtractError::WrongMediaType(media_type.to_string()));
- }
- };
-
- remove_replaced_paths(
- &mut VerifyingReader::new(decode_fn(&mut layer_file)?, layer_diffid)?,
- &target_path,
- )?;
-
- layer_file.seek(SeekFrom::Start(0))?;
- extract_archive(&mut decode_fn(&mut layer_file)?, &target_path)?;
-
- oci_tar_image = layer_file.into_oci_tar_image();
- }
-
- Ok(())
-}
-
-/// Validate a path previously received from a `tar::Entry`.
-///
-/// Tar files may contain arbitrary paths, so reject absolute paths and paths containing parent
-/// directory references, as joining them onto the target directory would lead outside of it.
-pub(crate) fn validate_path(path: &Path) -> io::Result<()> {
- use std::path::Component;
-
- if !path
- .components()
- .all(|c| matches!(c, Component::Normal(_) | Component::CurDir))
- {
- return Err(io::Error::other(format!(
- "illegal path {path:?} in layer - absolute paths and '..' components are not allowed"
- )));
- }
-
- Ok(())
-}
-
-/// Join the `path` of a layer entry onto the resolved `target_path`, resolving its parent only.
-///
-/// Fails if the parent leads outside of the target through a symlink, like the tar crate checks
-/// before unpacking, assuming that nothing else modifies the target meanwhile. Returns `None` if
-/// the parent does not exist or `path` has no last component.
-fn resolve_in_target(target_path: &Path, path: &Path) -> io::Result<Option<PathBuf>> {
- let (Some(parent), Some(name)) = (path.parent(), path.file_name()) else {
- return Ok(None);
- };
- Ok(resolve_dir_in_target(target_path, parent)?.map(|dir| dir.join(name)))
-}
-
-/// Resolve the directory `dir` of a layer entry within the resolved `target_path`, see
-/// [`resolve_in_target`].
-fn resolve_dir_in_target(target_path: &Path, dir: &Path) -> io::Result<Option<PathBuf>> {
- let resolved = match target_path.join(dir).canonicalize() {
- Ok(resolved) => resolved,
- Err(err) if matches!(err.kind(), ErrorKind::NotFound | ErrorKind::NotADirectory) => {
- return Ok(None);
- }
- Err(err) => return Err(err),
- };
-
- if !resolved.starts_with(target_path) {
- return Err(io::Error::other(format!(
- "illegal path {dir:?} in layer - a symlink leads outside of the target directory"
- )));
- }
-
- Ok(Some(resolved))
-}
-
-/// Remove what a layer replaces with directories or whites out from previous layers.
-///
-/// Reads `reader` to EOF before removing anything.
-fn remove_replaced_paths<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()> {
- let mut archive = Archive::new(reader);
-
- let mut directories = Vec::new();
- let mut whiteouts = Vec::new();
- for entry in archive.entries()? {
- let entry = entry?;
- let path = entry.path()?;
- validate_path(&path)?;
-
- match entry.header().entry_type() {
- EntryType::Directory => directories.push(path.into_owned()),
- EntryType::Regular if is_whiteout(&path) => whiteouts.push(path.into_owned()),
- _ => {}
- }
- }
-
- // Force an EOF so VerifyingReader actually verifies.
- io::copy(archive.into_inner(), &mut io::sink())?;
-
- // Parents first and before whiteouts, so nothing resolves through a symlink being replaced.
- directories.sort();
- for dir in directories {
- if let Some(dir_path_abs) = resolve_in_target(target_path, &dir)?
- && dir_path_abs.symlink_metadata().is_ok_and(|m| !m.is_dir())
- {
- remove_file(dir_path_abs)?;
- }
- }
-
- for whiteout in whiteouts {
- apply_whiteout(target_path, &whiteout)?;
- }
-
- Ok(())
-}
-
-fn is_whiteout(path: &Path) -> bool {
- path.file_name()
- .is_some_and(|name| name.as_bytes().starts_with(WHITEOUT_PREFIX))
-}
-
-/// Apply the whiteout file at `path` on previous layers.
-fn apply_whiteout(target_path: &Path, path: &Path) -> io::Result<()> {
- let Some(filename) = path.file_name() else {
- return Ok(());
- };
-
- if filename == OPAQUE_WHITEOUT_NAME {
- if let Some(parent) = path.parent()
- && let Some(whiteout_abs_path) = resolve_dir_in_target(target_path, parent)?
- {
- for direntry in read_dir(whiteout_abs_path)? {
- remove_path(&direntry?.path())?;
- }
- }
- } else if let Some(filename_stripped) = filename.as_bytes().strip_prefix(WHITEOUT_PREFIX) {
- let whiteout_path = match filename_stripped {
- b"." => match path.parent() {
- Some(p) if p.parent().is_some() => p,
- _ => return Ok(()), // Prevent whiteout of root directory
- },
- b".." => return Ok(()), // Prevent whiteout of grandparent directory
- fname => &path.with_file_name(OsStr::from_bytes(fname)),
- };
- if let Some(whiteout_abs_path) = resolve_in_target(target_path, whiteout_path)? {
- remove_path(&whiteout_abs_path)?;
- }
- }
-
- Ok(())
-}
-
-fn extract_archive<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()> {
- let mut archive = Archive::new(reader);
- archive.set_preserve_ownerships(true);
- archive.set_preserve_permissions(true);
- archive.set_unpack_xattrs(true);
-
- // Delay directory entries until the end (they will be created if needed by descendants),
- // to ensure that directory permissions do not interfere with descendant extraction.
- let mut directories = Vec::new();
- for entry in archive.entries()? {
- let mut file = entry?;
-
- let filepath = file.path()?;
- validate_path(&filepath)?;
-
- if file.header().entry_type() == EntryType::Directory {
- directories.push(file);
- continue;
- } else if file.header().entry_type() == EntryType::Regular && is_whiteout(&filepath) {
- continue;
- }
-
- // If a file or directory already exists at this path, remove it first.
- if let Some(file_path_abs) = resolve_in_target(target_path, &filepath)? {
- remove_path(&file_path_abs)?;
- }
-
- file.unpack_in(target_path)?;
- }
-
- // Apply the directories in reverse topological order,
- // to avoid failure on restrictive parent directory permissions.
- directories.sort_by(|a, b| b.path_bytes().cmp(&a.path_bytes()));
- for mut dir in directories {
- // Skip a directory replaced by a later entry, also to not apply it through a symlink.
- if let Some(dir_path_abs) = resolve_in_target(target_path, &dir.path()?)?
- && dir_path_abs.symlink_metadata().is_ok_and(|m| !m.is_dir())
- {
- continue;
- }
-
- dir.unpack_in(target_path)?;
- }
-
- Ok(())
-}
-
-/// Remove whatever exists at `path`, without following a symlink there.
-fn remove_path(path: &Path) -> io::Result<()> {
- match path.symlink_metadata() {
- Ok(metadata) if metadata.is_dir() => remove_dir_all(path),
- Ok(_) => remove_file(path),
- Err(err) if matches!(err.kind(), ErrorKind::NotFound | ErrorKind::NotADirectory) => Ok(()),
- Err(err) => Err(err),
- }
-}
diff --git a/proxmox-oci/src/parse.rs b/proxmox-oci/src/parse.rs
new file mode 100644
index 00000000..af669d38
--- /dev/null
+++ b/proxmox-oci/src/parse.rs
@@ -0,0 +1,52 @@
+use std::fs::File;
+use std::path::Path;
+
+use oci_spec::OciSpecError;
+use oci_spec::image::{Arch, ImageConfiguration, ImageManifest, MediaType, Os};
+use thiserror::Error;
+
+use crate::oci_tar_image::OciTarImage;
+
+#[derive(Debug, Error)]
+pub enum ParseError {
+ #[error(transparent)]
+ OciSpec(#[from] OciSpecError),
+ #[error("Wrong media type")]
+ WrongMediaType,
+ #[error(transparent)]
+ Io(#[from] std::io::Error),
+ #[error("No image manifest found in image index for the requested platform")]
+ NoMatchingManifest,
+ #[error("Missing image config")]
+ MissingImageConfig,
+}
+
+pub(crate) fn parse_image<P: AsRef<Path>>(
+ oci_tar_path: P,
+ os: Option<&Os>,
+ arch: Option<&Arch>,
+) -> Result<(OciTarImage<File>, ImageManifest, ImageConfiguration), ParseError> {
+ let oci_tar_file = File::open(oci_tar_path)?;
+ let mut oci_tar_image = OciTarImage::new(oci_tar_file)?;
+
+ let image_manifest = oci_tar_image
+ .image_manifest(os, arch)
+ .ok_or(ParseError::NoMatchingManifest)??;
+
+ let image_config_descriptor = image_manifest.config();
+
+ if image_config_descriptor.media_type() != &MediaType::ImageConfig {
+ return Err(ParseError::WrongMediaType);
+ }
+
+ let mut image_config_file = oci_tar_image
+ .open_blob(image_config_descriptor.digest())
+ .ok_or(ParseError::MissingImageConfig)?;
+ let image_config = ImageConfiguration::from_reader(&mut image_config_file)?;
+
+ Ok((
+ image_config_file.into_oci_tar_image(),
+ image_manifest,
+ image_config,
+ ))
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 08/26] oci: explicitly set mtime of directories when extracting layer
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (6 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 07/26] oci: factor out parsing and extraction into modules Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 09/26] oci: add a registry client Filip Schauer
` (17 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
The tar crate doesn't set the mtime of directories when unpacking a TAR
archive, even though preserving mtimes is enabled by default. Therefore
set the mtime manually.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
| 25 ++++++++++++++++++++++---
1 file changed, 22 insertions(+), 3 deletions(-)
--git a/proxmox-oci/src/extract.rs b/proxmox-oci/src/extract.rs
index d9d5f1b4..b158c3be 100644
--- a/proxmox-oci/src/extract.rs
+++ b/proxmox-oci/src/extract.rs
@@ -1,9 +1,10 @@
use std::ffi::OsStr;
-use std::fs::{read_dir, remove_dir_all, remove_file};
+use std::fs::{File, read_dir, remove_dir_all, remove_file};
use std::io::{self, ErrorKind, Read, Seek, SeekFrom};
use std::os::unix::ffi::OsStrExt;
use std::path::{Path, PathBuf};
use std::str::FromStr;
+use std::time::{Duration, UNIX_EPOCH};
use flate2::read::GzDecoder;
use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType};
@@ -249,14 +250,32 @@ fn extract_archive<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()
// to avoid failure on restrictive parent directory permissions.
directories.sort_by(|a, b| b.path_bytes().cmp(&a.path_bytes()));
for mut dir in directories {
+ let dir_path = dir.path()?.into_owned();
+
// Skip a directory replaced by a later entry, also to not apply it through a symlink.
- if let Some(dir_path_abs) = resolve_in_target(target_path, &dir.path()?)?
+ if let Some(dir_path_abs) = resolve_in_target(target_path, &dir_path)?
&& dir_path_abs.symlink_metadata().is_ok_and(|m| !m.is_dir())
{
continue;
}
- dir.unpack_in(target_path)?;
+ if !dir.unpack_in(target_path)? {
+ continue;
+ }
+
+ // Manually set the mtime of the directory, since tar-rs doesn't do it.
+ if let Ok(mtime) = dir.header().mtime()
+ && let Some(dir_path_abs) = resolve_dir_in_target(target_path, &dir_path)?
+ {
+ let secs = mtime.cast_signed();
+ let duration = Duration::from_secs(secs.unsigned_abs());
+ let mtime = if secs > 0 {
+ UNIX_EPOCH + duration
+ } else {
+ UNIX_EPOCH - duration
+ };
+ File::open(dir_path_abs)?.set_modified(mtime)?;
+ }
}
Ok(())
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 09/26] oci: add a registry client
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (7 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 08/26] oci: explicitly set mtime of directories when extracting layer Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 10/26] oci: add a public list_repository_tags function Filip Schauer
` (16 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
The client code is inspired by
https://github.com/oras-project/rust-oci-client
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/Cargo.toml | 7 +
proxmox-oci/debian/control | 16 +
proxmox-oci/src/client.rs | 612 +++++++++++++++++++++++++++++++++++++
proxmox-oci/src/digest.rs | 19 ++
proxmox-oci/src/lib.rs | 2 +
5 files changed, 656 insertions(+)
create mode 100644 proxmox-oci/src/client.rs
diff --git a/proxmox-oci/Cargo.toml b/proxmox-oci/Cargo.toml
index 08c47db5..45248658 100644
--- a/proxmox-oci/Cargo.toml
+++ b/proxmox-oci/Cargo.toml
@@ -13,12 +13,19 @@ rust-version.workspace = true
[dependencies]
flate2.workspace = true
+http.workspace = true
+http-auth = { version = "0.1", default-features = false }
oci-spec = "0.8.3"
+serde.workspace = true
+serde_json.workspace = true
sha2 = "0.10"
tar.workspace = true
thiserror = "1"
+url.workspace = true
zstd.workspace = true
+proxmox-base64.workspace = true
+proxmox-http = { workspace = true, features = [ "client-sync" ] }
proxmox-io.workspace = true
[dev-dependencies]
diff --git a/proxmox-oci/debian/control b/proxmox-oci/debian/control
index a742b948..9ec73059 100644
--- a/proxmox-oci/debian/control
+++ b/proxmox-oci/debian/control
@@ -7,11 +7,19 @@ Build-Depends-Arch: cargo:native <!nocheck>,
rustc:native (>= 1.88) <!nocheck>,
libstd-rust-dev <!nocheck>,
librust-flate2-1+default-dev <!nocheck>,
+ librust-http-1+default-dev <!nocheck>,
+ librust-http-auth-0.1-dev <!nocheck>,
librust-oci-spec-0.8+default-dev (>= 0.8.3-~~) <!nocheck>,
+ librust-proxmox-base64-1+default-dev <!nocheck>,
+ librust-proxmox-http-1+client-sync-dev (>= 1.0.5-~~) <!nocheck>,
+ librust-proxmox-http-1+default-dev (>= 1.0.5-~~) <!nocheck>,
librust-proxmox-io-1+default-dev (>= 1.2.1-~~) <!nocheck>,
+ librust-serde-1+default-dev <!nocheck>,
+ librust-serde-json-1+default-dev <!nocheck>,
librust-sha2-0.10+default-dev <!nocheck>,
librust-tar-0.4+default-dev <!nocheck>,
librust-thiserror-1+default-dev <!nocheck>,
+ librust-url-2+default-dev (>= 2.2-~~) <!nocheck>,
librust-zstd-0.13+default-dev <!nocheck>
Maintainer: Proxmox Support Team <support@proxmox.com>
Standards-Version: 4.7.2
@@ -26,11 +34,19 @@ Multi-Arch: same
Depends:
${misc:Depends},
librust-flate2-1+default-dev,
+ librust-http-1+default-dev,
+ librust-http-auth-0.1-dev,
librust-oci-spec-0.8+default-dev (>= 0.8.3-~~),
+ librust-proxmox-base64-1+default-dev,
+ librust-proxmox-http-1+client-sync-dev (>= 1.0.5-~~),
+ librust-proxmox-http-1+default-dev (>= 1.0.5-~~),
librust-proxmox-io-1+default-dev (>= 1.2.1-~~),
+ librust-serde-1+default-dev,
+ librust-serde-json-1+default-dev,
librust-sha2-0.10+default-dev,
librust-tar-0.4+default-dev,
librust-thiserror-1+default-dev,
+ librust-url-2+default-dev (>= 2.2-~~),
librust-zstd-0.13+default-dev
Provides:
librust-proxmox-oci+default-dev (= ${binary:Version}),
diff --git a/proxmox-oci/src/client.rs b/proxmox-oci/src/client.rs
new file mode 100644
index 00000000..c6d15cbc
--- /dev/null
+++ b/proxmox-oci/src/client.rs
@@ -0,0 +1,612 @@
+use std::collections::HashMap;
+use std::io::{self, Cursor, Read, Write};
+use std::str::FromStr;
+use std::time::{Duration, Instant};
+
+use http::HeaderValue;
+use http::uri::Scheme;
+use http::{HeaderMap, StatusCode};
+use http_auth::{ChallengeParser, ChallengeRef};
+use oci_spec::OciSpecError;
+use oci_spec::distribution::{ErrorResponse, Reference, TagList};
+use oci_spec::image::{
+ Arch, Descriptor, Digest, ImageConfiguration, ImageIndex, ImageManifest, MediaType, Os,
+};
+use proxmox_http::client::sync;
+use proxmox_http::{HttpClient, HttpOptions, ProxyConfig};
+use serde::Deserialize;
+use thiserror::Error;
+use url::Url;
+
+use crate::digest::{DigestError, VerifyingReader, verify_digest};
+
+// Inspired by https://github.com/oras-project/rust-oci-client
+
+#[derive(Deserialize)]
+pub enum RegistryAuth {
+ /// Access the registry anonymously
+ Anonymous,
+ /// Access the registry using HTTP Basic authentication
+ Basic(String, String),
+ /// Access the registry using Bearer token authentication
+ Bearer(String),
+}
+
+struct BearerChallenge {
+ realm: Url,
+ service: Option<String>,
+}
+
+impl TryFrom<&HeaderValue> for BearerChallenge {
+ type Error = String;
+
+ fn try_from(value: &HeaderValue) -> Result<Self, Self::Error> {
+ let header = value
+ .to_str()
+ .map_err(|e| format!("cannot convert header value to string: {e:?}"))?;
+ ChallengeParser::new(header)
+ .filter_map(Result::ok)
+ .find_map(|c| BearerChallenge::try_from(&c).ok())
+ .ok_or_else(|| "Cannot find Bearer challenge".to_string())
+ }
+}
+
+impl TryFrom<&ChallengeRef<'_>> for BearerChallenge {
+ type Error = String;
+
+ fn try_from(value: &ChallengeRef<'_>) -> Result<Self, Self::Error> {
+ if !value.scheme.eq_ignore_ascii_case("Bearer") {
+ return Err(format!(
+ "BearerChallenge doesn't support challenge scheme {:?}",
+ value.scheme
+ ));
+ }
+
+ let mut realm = None;
+ let mut service = None;
+
+ for (k, v) in &value.params {
+ if k.eq_ignore_ascii_case("realm") {
+ realm = Some(Url::parse(&v.to_unescaped()).map_err(|err| err.to_string())?);
+ }
+ if k.eq_ignore_ascii_case("service") {
+ service = Some(v.to_unescaped());
+ }
+ }
+
+ if let Some(realm) = realm {
+ Ok(BearerChallenge { realm, service })
+ } else {
+ Err("missing required parameter realm".to_string())
+ }
+ }
+}
+
+enum RegistryTokenType {
+ Bearer(String),
+ Basic(String, String),
+}
+
+struct TokenCacheValue {
+ token: RegistryTokenType,
+ expiration: Option<Instant>,
+}
+
+struct TokenCache(HashMap<Reference, TokenCacheValue>);
+
+impl TokenCache {
+ fn new() -> Self {
+ Self(HashMap::new())
+ }
+
+ fn insert(&mut self, reference: Reference, token: RegistryTokenType) {
+ let expiration = match token {
+ RegistryTokenType::Basic(_, _) => None,
+ RegistryTokenType::Bearer(_) => Instant::now().checked_add(Duration::from_secs(60)),
+ };
+ self.0
+ .insert(reference, TokenCacheValue { token, expiration });
+ }
+
+ fn get(&self, reference: &Reference) -> Option<&RegistryTokenType> {
+ let TokenCacheValue { token, expiration } = self.0.get(reference)?;
+ expiration
+ .is_none_or(|t| t > Instant::now())
+ .then_some(token)
+ }
+}
+
+#[derive(Error, Debug)]
+pub enum OciClientError {
+ #[error("Authentication failure: {0}")]
+ AuthenticationFailure(String),
+ #[error(transparent)]
+ Digest(#[from] DigestError),
+ #[error(transparent)]
+ Io(#[from] io::Error),
+ #[error("No image manifest found in image index for the requested platform")]
+ NoMatchingManifest,
+ #[error("Registry error: url {url}, envelope: {envelope}")]
+ Registry {
+ /// List of errors returned by the OCI registry
+ envelope: ErrorResponse,
+ /// Request URL
+ url: Url,
+ },
+ #[error("Transport error: {0}")]
+ Transport(String),
+ #[error("Server error: url {url}, code: {code}, message: {message}")]
+ Server {
+ code: http::StatusCode,
+ url: Url,
+ message: String,
+ },
+ #[error("OCI distribution spec violation: {0}")]
+ SpecViolation(String),
+ #[error("Not authorized: url {url}")]
+ Unauthorized { url: Url },
+ #[error(transparent)]
+ UrlParse(#[from] url::ParseError),
+ #[error(transparent)]
+ OciSpec(#[from] OciSpecError),
+}
+
+pub struct DigestedImageManifest {
+ pub manifest: ImageManifest,
+ pub raw: Vec<u8>,
+ pub digest: Digest,
+}
+
+pub struct DigestedImageConfig {
+ pub config: ImageConfiguration,
+ pub raw: Vec<u8>,
+ pub digest: Digest,
+}
+
+pub struct ImageMetadata {
+ pub manifest: DigestedImageManifest,
+ pub config: DigestedImageConfig,
+}
+
+struct OciResponse {
+ status: StatusCode,
+ headers: HeaderMap,
+ url: Url,
+ body: Box<dyn Read>,
+}
+
+impl OciResponse {
+ fn into_body(self) -> Box<dyn Read> {
+ self.body
+ }
+
+ fn read_to_end(mut self) -> Result<Vec<u8>, OciClientError> {
+ let mut buf = Vec::new();
+ self.body.read_to_end(&mut buf)?;
+ Ok(buf)
+ }
+
+ fn read_to_string(mut self) -> Result<String, OciClientError> {
+ let mut s = String::new();
+ self.body.read_to_string(&mut s)?;
+ Ok(s)
+ }
+}
+
+struct OciRequest {
+ url: Url,
+ headers: Vec<(String, String)>,
+ query_params: Vec<(String, String)>,
+}
+
+impl OciRequest {
+ fn new(url: Url) -> Self {
+ Self {
+ url,
+ headers: Vec::new(),
+ query_params: Vec::new(),
+ }
+ }
+
+ fn header(mut self, name: &str, value: &str) -> Self {
+ self.headers.push((name.to_string(), value.to_string()));
+ self
+ }
+
+ fn query(mut self, key: &str, value: &str) -> Self {
+ self.query_params.push((key.to_string(), value.to_string()));
+ self
+ }
+
+ fn with_accept(mut self, accept: &[&str]) -> Self {
+ self.headers.push(("Accept".to_string(), accept.join(", ")));
+ self
+ }
+
+ fn with_auth(mut self, token: &RegistryTokenType) -> Self {
+ let value = match token {
+ RegistryTokenType::Bearer(token) => format!("Bearer {token}"),
+ RegistryTokenType::Basic(username, password) => {
+ let creds = proxmox_base64::encode(format!("{username}:{password}"));
+ format!("Basic {creds}")
+ }
+ };
+ self.headers.push(("Authorization".to_string(), value));
+ self
+ }
+
+ fn build_url(&self) -> Url {
+ let mut url = self.url.clone();
+
+ for (k, v) in &self.query_params {
+ url.query_pairs_mut().append_pair(k, v);
+ }
+
+ url
+ }
+}
+
+pub struct ClientConfig {
+ pub protocol: Scheme,
+ pub timeout: Option<Duration>,
+ pub proxy_config: Option<ProxyConfig>,
+}
+
+impl Default for ClientConfig {
+ fn default() -> Self {
+ Self {
+ protocol: Scheme::HTTPS,
+ timeout: Some(Duration::from_secs(20)),
+ proxy_config: None,
+ }
+ }
+}
+
+pub struct Client {
+ protocol: Scheme,
+ auth: RegistryAuth,
+ tokens: TokenCache,
+ http_client: sync::Client,
+}
+
+impl Client {
+ pub fn new(config: ClientConfig, auth: RegistryAuth) -> Self {
+ let options = HttpOptions {
+ proxy_config: config.proxy_config,
+ ..Default::default()
+ };
+
+ let http_client = if let Some(timeout) = config.timeout {
+ sync::Client::new_with_timeout(options, timeout)
+ } else {
+ sync::Client::new(options)
+ }
+ .with_http_status_as_error(false);
+
+ Self {
+ tokens: TokenCache::new(),
+ protocol: config.protocol,
+ auth,
+ http_client,
+ }
+ }
+
+ fn get_auth_token(
+ &mut self,
+ reference: &Reference,
+ ) -> Result<Option<&RegistryTokenType>, OciClientError> {
+ if self.tokens.get(reference).is_none() {
+ match self.authenticate(reference, &self.auth)? {
+ Some(token) => self.tokens.insert(reference.clone(), token),
+ None => return Ok(None),
+ };
+ }
+
+ Ok(self.tokens.get(reference))
+ }
+
+ fn authenticate(
+ &self,
+ image: &Reference,
+ authentication: &RegistryAuth,
+ ) -> Result<Option<RegistryTokenType>, OciClientError> {
+ if let RegistryAuth::Bearer(token) = authentication {
+ return Ok(Some(RegistryTokenType::Bearer(token.clone())));
+ }
+
+ let url = Url::parse(&format!(
+ "{}://{}/v2/",
+ self.protocol,
+ image.resolve_registry()
+ ))?;
+ let req = OciRequest::new(url);
+ let res = self.send(req)?;
+ let dist_hdr = match res.headers.get(http::header::WWW_AUTHENTICATE) {
+ Some(h) => h,
+ None => return Ok(None),
+ };
+
+ let challenge = match BearerChallenge::try_from(dist_hdr) {
+ Ok(c) => c,
+ Err(_) => {
+ if let RegistryAuth::Basic(username, password) = authentication {
+ return Ok(Some(RegistryTokenType::Basic(
+ username.to_string(),
+ password.to_string(),
+ )));
+ }
+ return Ok(None);
+ }
+ };
+
+ let mut auth_req = OciRequest::new(challenge.realm)
+ .query("scope", &format!("repository:{}:pull", image.repository()));
+
+ if let Some(s) = challenge.service {
+ auth_req = auth_req.query("service", &s);
+ }
+
+ if let RegistryAuth::Basic(username, password) = authentication {
+ let creds = proxmox_base64::encode(format!("{username}:{password}"));
+ auth_req = auth_req.header("Authorization", &format!("Basic {creds}"));
+ }
+
+ let auth_res = self.send(auth_req)?;
+ let status = auth_res.status;
+ let text = auth_res.read_to_string()?;
+
+ if status != StatusCode::OK {
+ return Err(OciClientError::AuthenticationFailure(text));
+ }
+
+ serde_json::from_str::<serde_json::Value>(&text)
+ .ok()
+ .as_ref()
+ .and_then(|j| j["token"].as_str().or(j["access_token"].as_str()))
+ .map(|t| Some(RegistryTokenType::Bearer(t.into())))
+ .ok_or_else(|| {
+ OciClientError::AuthenticationFailure("Invalid response or missing token".into())
+ })
+ }
+
+ pub fn list_tags(
+ &mut self,
+ image: &Reference,
+ n: Option<usize>,
+ last: Option<&str>,
+ ) -> Result<TagList, OciClientError> {
+ let url = self.to_v2_url(image, "tags/list")?;
+ let mut req = OciRequest::new(url.clone());
+ if let Some(num) = n {
+ req = req.query("n", &num.to_string());
+ }
+ if let Some(l) = last {
+ req = req.query("last", l);
+ }
+
+ if let Some(auth_token) = self.get_auth_token(image)? {
+ req = req.with_auth(auth_token);
+ }
+ let res = self.send(req)?;
+ let status = res.status;
+ let body = res.read_to_end()?;
+ validate_registry_response(status, &body, &url)?;
+ Ok(serde_json::from_slice(&body).map_err(OciSpecError::SerDe)?)
+ }
+
+ pub fn pull_image_manifest(
+ &mut self,
+ image: &Reference,
+ os: Option<&Os>,
+ arch: Option<&Arch>,
+ ) -> Result<DigestedImageManifest, OciClientError> {
+ let (body, digest, media_type) = self.pull_manifest_raw(
+ image,
+ &[
+ MediaType::ImageManifest.as_ref(),
+ MediaType::ImageIndex.as_ref(),
+ ],
+ )?;
+
+ if matches!(media_type, MediaType::ImageIndex) {
+ let index = ImageIndex::from_reader(Cursor::new(&body))?;
+
+ let manifest_digest = index
+ .manifests()
+ .iter()
+ .find(|m| {
+ m.platform().as_ref().is_none_or(|p| {
+ arch.is_none_or(|a| p.architecture() == a) && os.is_none_or(|o| p.os() == o)
+ })
+ })
+ .map(|m| m.digest().clone())
+ .ok_or(OciClientError::NoMatchingManifest)?;
+
+ let manifest_reference = image.clone_with_digest(manifest_digest.to_string());
+ let (manifest_raw, manifest_digest, _) =
+ self.pull_manifest_raw(&manifest_reference, &[MediaType::ImageManifest.as_ref()])?;
+ let manifest = ImageManifest::from_reader(Cursor::new(&manifest_raw))?;
+
+ Ok(DigestedImageManifest {
+ manifest,
+ raw: manifest_raw,
+ digest: manifest_digest,
+ })
+ } else {
+ let manifest = ImageManifest::from_reader(Cursor::new(&body))?;
+ Ok(DigestedImageManifest {
+ manifest,
+ raw: body,
+ digest,
+ })
+ }
+ }
+
+ pub fn pull_manifest_raw(
+ &mut self,
+ image: &Reference,
+ accepted_media_types: &[&str],
+ ) -> Result<(Vec<u8>, Digest, MediaType), OciClientError> {
+ let url = self.to_v2_url(
+ image,
+ &format!(
+ "manifests/{}",
+ image.digest().or_else(|| image.tag()).unwrap_or("latest")
+ ),
+ )?;
+
+ let mut req = OciRequest::new(url).with_accept(accepted_media_types);
+ if let Some(auth_token) = self.get_auth_token(image)? {
+ req = req.with_auth(auth_token);
+ }
+ let res = self.send(req)?;
+
+ let status = res.status;
+ let url = res.url.clone();
+ let headers = res.headers.clone();
+ let body = res.read_to_end()?;
+ validate_registry_response(status, &body, &url)?;
+
+ let expected_digest = image.digest().map(Digest::from_str).transpose()?;
+ let digest = verify_digest(&body, expected_digest)?;
+
+ let Some(media_type) = headers
+ .get(http::header::CONTENT_TYPE)
+ .and_then(|v| v.to_str().ok())
+ .map(MediaType::from)
+ else {
+ return Err(OciClientError::SpecViolation("Missing Content-Type".into()));
+ };
+
+ Ok((body, digest, media_type))
+ }
+
+ pub fn pull_image_manifest_and_config(
+ &mut self,
+ image: &Reference,
+ os: Option<&Os>,
+ arch: Option<&Arch>,
+ ) -> Result<ImageMetadata, OciClientError> {
+ let digested_manifest = self.pull_image_manifest(image, os, arch)?;
+
+ let config_descriptor = digested_manifest.manifest.config();
+ let mut config_bytes = Vec::new();
+ self.pull_blob(image, config_descriptor, &mut config_bytes)?;
+
+ let config = ImageConfiguration::from_reader(Cursor::new(&config_bytes))?;
+ let digested_config = DigestedImageConfig {
+ config,
+ raw: config_bytes,
+ digest: config_descriptor.digest().clone(),
+ };
+
+ Ok(ImageMetadata {
+ manifest: digested_manifest,
+ config: digested_config,
+ })
+ }
+
+ pub fn pull_blob<W: Write>(
+ &mut self,
+ image: &Reference,
+ layer: &Descriptor,
+ out: &mut W,
+ ) -> Result<(), OciClientError> {
+ let mut reader = self.pull_blob_reader(image, layer)?;
+ io::copy(&mut reader, out).map_err(|err| {
+ err.downcast::<DigestError>()
+ .map(OciClientError::Digest)
+ .unwrap_or_else(OciClientError::Io)
+ })?;
+ out.flush()?;
+
+ Ok(())
+ }
+
+ pub fn pull_blob_reader(
+ &mut self,
+ image: &Reference,
+ layer: &Descriptor,
+ ) -> Result<VerifyingReader<Box<dyn Read>>, OciClientError> {
+ let url = self.to_v2_url(image, &format!("blobs/{}", layer.digest()))?;
+
+ let mut request = OciRequest::new(url);
+ if let Some(auth_token) = self.get_auth_token(image)? {
+ request = request.with_auth(auth_token);
+ }
+
+ let response = self.send(request)?;
+ let status = response.status;
+ let url = response.url.clone();
+
+ if !status.is_success() {
+ let body = response.read_to_end()?;
+ return Err(validate_registry_response(status, &body, &url)
+ .expect_err("non-success status code must produce error"));
+ }
+
+ Ok(VerifyingReader::new(
+ response.into_body(),
+ layer.digest().clone(),
+ )?)
+ }
+
+ fn to_v2_url(&self, reference: &Reference, suffix: &str) -> Result<Url, url::ParseError> {
+ Url::parse(&format!(
+ "{}://{}/v2/{}/{suffix}",
+ self.protocol,
+ reference.resolve_registry(),
+ reference.repository(),
+ ))
+ }
+
+ fn send(&self, req: OciRequest) -> Result<OciResponse, OciClientError> {
+ let url = req.build_url();
+
+ let headers = req.headers.into_iter().collect();
+
+ let res = self
+ .http_client
+ .get(url.as_str(), Some(&headers))
+ .map_err(|e| OciClientError::Transport(e.to_string()))?;
+
+ let (parts, body) = res.into_parts();
+
+ Ok(OciResponse {
+ status: parts.status,
+ headers: parts.headers,
+ url,
+ body,
+ })
+ }
+}
+
+fn validate_registry_response(
+ status: StatusCode,
+ body: &[u8],
+ url: &Url,
+) -> Result<(), OciClientError> {
+ match status {
+ StatusCode::OK => Ok(()),
+ StatusCode::UNAUTHORIZED => Err(OciClientError::Unauthorized { url: url.clone() }),
+ s if s.is_success() => Err(OciClientError::SpecViolation(format!(
+ "Expected HTTP Status 200 OK, got {status} instead"
+ ))),
+ s if s.is_client_error() => match serde_json::from_slice::<ErrorResponse>(body) {
+ Ok(envelope) => Err(OciClientError::Registry {
+ envelope,
+ url: url.clone(),
+ }),
+ Err(_) => Err(OciClientError::Server {
+ code: s,
+ url: url.clone(),
+ message: String::from_utf8_lossy(body).into_owned(),
+ }),
+ },
+ s => Err(OciClientError::Server {
+ code: s,
+ url: url.clone(),
+ message: String::from_utf8_lossy(body).into_owned(),
+ }),
+ }
+}
diff --git a/proxmox-oci/src/digest.rs b/proxmox-oci/src/digest.rs
index b2170b87..af264f06 100644
--- a/proxmox-oci/src/digest.rs
+++ b/proxmox-oci/src/digest.rs
@@ -43,6 +43,25 @@ impl Digester {
}
}
+pub fn verify_digest(body: &[u8], expected_digest: Option<Digest>) -> Result<Digest, DigestError> {
+ let algorithm = expected_digest
+ .as_ref()
+ .map(|d| d.algorithm())
+ .unwrap_or(&DigestAlgorithm::Sha256);
+
+ let mut digester = Digester::new(algorithm)?;
+ digester.update(body);
+ let actual = digester.finalize();
+
+ if let Some(expected) = expected_digest
+ && actual != expected
+ {
+ return Err(DigestError::Mismatch { expected, actual });
+ }
+
+ Ok(actual)
+}
+
pub struct VerifyingReader<R: Read> {
inner: R,
digester: Digester,
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index fcb81b03..274563f4 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -3,6 +3,8 @@ use std::path::Path;
pub use oci_spec::image::{Arch, Config, Os};
use thiserror::Error;
+pub mod client;
+
mod digest;
mod extract;
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 10/26] oci: add a public list_repository_tags function
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (8 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 09/26] oci: add a registry client Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 11/26] pve-rs: oci: expose list_repository_tags Filip Schauer
` (15 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/lib.rs | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index 274563f4..79e286a4 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -1,9 +1,11 @@
use std::path::Path;
+pub use oci_spec::distribution::Reference;
pub use oci_spec::image::{Arch, Config, Os};
use thiserror::Error;
pub mod client;
+use client::{Client, ClientConfig, OciClientError, RegistryAuth};
mod digest;
@@ -52,3 +54,18 @@ pub fn parse_and_extract_image<P: AsRef<Path>>(
Ok(image_config.config().clone())
}
+
+/// List all tags available for a repository in an OCI registry.
+///
+/// # Arguments
+///
+/// * `reference` - OCI reference specifying the target registry and repository
+/// * `auth` - Authentication credentials/method to access the registry
+pub fn list_repository_tags(
+ reference: &Reference,
+ auth: RegistryAuth,
+) -> Result<Vec<String>, OciClientError> {
+ let mut client = Client::new(ClientConfig::default(), auth);
+ let tag_list = client.list_tags(reference, None, None)?;
+ Ok(tag_list.tags().to_vec())
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox-perl-rs 11/26] pve-rs: oci: expose list_repository_tags
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (9 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 10/26] oci: add a public list_repository_tags function Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH manager 12/26] use proxmox-oci instead of skopeo for listing tags from OCI registries Filip Schauer
` (14 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
pve-rs/src/bindings/oci.rs | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/pve-rs/src/bindings/oci.rs b/pve-rs/src/bindings/oci.rs
index 11ac07f..51bf29d 100644
--- a/pve-rs/src/bindings/oci.rs
+++ b/pve-rs/src/bindings/oci.rs
@@ -4,8 +4,11 @@ pub mod pve_rs_oci {
//!
//! Provides bindings for the [`proxmox_oci`] crate.
+ use std::str::FromStr;
+
use anyhow::Error;
- use proxmox_oci::Config;
+ use proxmox_oci::client::RegistryAuth;
+ use proxmox_oci::{Config, Reference};
/// Extract the rootfs of an OCI image tar and return its execution parameters.
///
@@ -39,4 +42,16 @@ pub mod pve_rs_oci {
.map(|config| config.unwrap_or_default())
.map_err(Into::into)
}
+
+ /// List all tags available for a repository in an OCI registry.
+ ///
+ /// # Arguments
+ ///
+ /// * `reference` - OCI reference specifying the target registry and repository
+ /// * `auth` - Authentication credentials/method to access the registry
+ #[export]
+ pub fn list_repository_tags(reference: &str, auth: RegistryAuth) -> Result<Vec<String>, Error> {
+ let reference = Reference::from_str(reference)?;
+ Ok(proxmox_oci::list_repository_tags(&reference, auth)?)
+ }
}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH manager 12/26] use proxmox-oci instead of skopeo for listing tags from OCI registries
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (10 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 11/26] pve-rs: oci: expose list_repository_tags Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 13/26] oci: rename parse_and_extract_image to be tar-specific Filip Schauer
` (13 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
PVE/API2/Nodes.pm | 13 ++-----------
debian/control | 1 -
2 files changed, 2 insertions(+), 12 deletions(-)
diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
index 1e2a96f1..6a7eeb86 100644
--- a/PVE/API2/Nodes.pm
+++ b/PVE/API2/Nodes.pm
@@ -1905,18 +1905,9 @@ __PACKAGE__->register_method({
code => sub {
my ($param) = @_;
- die "Install 'skopeo' to list tags from OCI registries.\n" if (!-f '/usr/bin/skopeo');
-
my $reference = $param->{reference};
- my $tags_json = "";
- PVE::Tools::run_command(
- ["skopeo", "list-tags", "docker://$reference"],
- outfunc => sub {
- $tags_json = $tags_json . shift;
- },
- );
- my $tags = decode_json($tags_json);
- return $tags->{Tags};
+ my $auth = "Anonymous";
+ return PVE::RS::OCI::list_repository_tags($reference, $auth);
},
});
diff --git a/debian/control b/debian/control
index e438ce12..7d3de1d8 100644
--- a/debian/control
+++ b/debian/control
@@ -107,7 +107,6 @@ Depends: apt (>= 1.5~),
Recommends: proxmox-firewall,
proxmox-offline-mirror-helper,
pve-nvidia-vgpu-helper,
- skopeo,
Conflicts: vlan, vzdump,
Replaces: vlan, vzdump,
Provides: vlan, vzdump,
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 13/26] oci: rename parse_and_extract_image to be tar-specific
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (11 preceding siblings ...)
2026-11-12 14:10 ` [PATCH manager 12/26] use proxmox-oci instead of skopeo for listing tags from OCI registries Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 14/26] pve-rs: " Filip Schauer
` (12 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Rename `parse_and_extract_image` to `parse_and_extract_tar_image` to
clarify that the function specifically operates on an OCI image in a TAR
archive.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/lib.rs | 2 +-
proxmox-oci/tests/helper/mod.rs | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index 79e286a4..5652a50f 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -42,7 +42,7 @@ pub enum ProxmoxOciError {
///
/// On success, this returns the optional container execution parameters
/// if present in the image configuration.
-pub fn parse_and_extract_image<P: AsRef<Path>>(
+pub fn parse_and_extract_tar_image<P: AsRef<Path>>(
oci_tar_path: P,
rootfs_path: P,
os: Option<&Os>,
diff --git a/proxmox-oci/tests/helper/mod.rs b/proxmox-oci/tests/helper/mod.rs
index 46491a89..56e51fda 100644
--- a/proxmox-oci/tests/helper/mod.rs
+++ b/proxmox-oci/tests/helper/mod.rs
@@ -5,7 +5,7 @@ use std::path::{Path, PathBuf};
use anyhow::Result;
-use proxmox_oci::{Arch, Config, Os, ProxmoxOciError, parse_and_extract_image};
+use proxmox_oci::{Arch, Config, Os, ProxmoxOciError, parse_and_extract_tar_image};
use proxmox_sys::fs::make_tmp_dir;
mod oci_tar_image_builder;
@@ -71,7 +71,7 @@ impl TestImage {
}
pub fn extract(&self) -> Result<Option<Config>, ProxmoxOciError> {
- parse_and_extract_image(
+ parse_and_extract_tar_image(
&self.path,
&self.extract_dir,
Some(&Os::Linux),
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox-perl-rs 14/26] pve-rs: oci: rename parse_and_extract_image to be tar-specific
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (12 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 13/26] oci: rename parse_and_extract_image to be tar-specific Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH container 15/26] oci create: " Filip Schauer
` (11 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Rename `parse_and_extract_image` to `parse_and_extract_tar_image` to
clarify that the function specifically operates on an OCI image in a TAR
archive.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
pve-rs/src/bindings/oci.rs | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pve-rs/src/bindings/oci.rs b/pve-rs/src/bindings/oci.rs
index 51bf29d..e26f336 100644
--- a/pve-rs/src/bindings/oci.rs
+++ b/pve-rs/src/bindings/oci.rs
@@ -26,14 +26,14 @@ pub mod pve_rs_oci {
/// On success, this returns the optional container execution parameters
/// if present in the image configuration.
#[export]
- pub fn parse_and_extract_image(
+ pub fn parse_and_extract_tar_image(
oci_tar_path: &str,
rootfs_path: &str,
os: Option<&str>,
arch: Option<&str>,
) -> Result<Config, Error> {
let arch = arch.map(Into::into);
- proxmox_oci::parse_and_extract_image(
+ proxmox_oci::parse_and_extract_tar_image(
oci_tar_path,
rootfs_path,
os.map(Into::into).as_ref(),
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH container 15/26] oci create: rename parse_and_extract_image to be tar-specific
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (13 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 14/26] pve-rs: " Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 16/26] oci: rename ProxmoxOciError to ParseAndExtractTarError Filip Schauer
` (10 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Rename `parse_and_extract_image` to `parse_and_extract_tar_image` to
clarify that the function specifically operates on an OCI image in a TAR
archive.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/LXC/Create.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/PVE/LXC/Create.pm b/src/PVE/LXC/Create.pm
index 64a6f66..49b04e9 100644
--- a/src/PVE/LXC/Create.pm
+++ b/src/PVE/LXC/Create.pm
@@ -679,12 +679,12 @@ sub restore_oci_archive {
if (@$id_map) {
$unsafe_oci_config = PVE::LXC::Namespaces::run_in_userns(
sub {
- PVE::RS::OCI::parse_and_extract_image($archive_file, $rootdir);
+ PVE::RS::OCI::parse_and_extract_tar_image($archive_file, $rootdir);
},
$id_map,
);
} else {
- $unsafe_oci_config = PVE::RS::OCI::parse_and_extract_image($archive_file, $rootdir);
+ $unsafe_oci_config = PVE::RS::OCI::parse_and_extract_tar_image($archive_file, $rootdir);
}
# should we rather validate this on the rust side already?
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 16/26] oci: rename ProxmoxOciError to ParseAndExtractTarError
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (14 preceding siblings ...)
2026-11-12 14:10 ` [PATCH container 15/26] oci create: " Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 17/26] oci: add a public pull_image_as_tar function Filip Schauer
` (9 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Rename `ProxmoxOciError` to `ParseAndExtractTarError` to match
`parse_and_extract_tar_image`, in preparation for introducing dedicated
error types for other operations.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/lib.rs | 4 ++--
proxmox-oci/tests/helper/mod.rs | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index 5652a50f..672d81e0 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -20,7 +20,7 @@ use parse::parse_image;
mod oci_tar_image;
#[derive(Debug, Error)]
-pub enum ProxmoxOciError {
+pub enum ParseAndExtractTarError {
#[error("Error while parsing OCI image: {0}")]
ParseError(#[from] ParseError),
#[error("Error while extracting OCI image: {0}")]
@@ -47,7 +47,7 @@ pub fn parse_and_extract_tar_image<P: AsRef<Path>>(
rootfs_path: P,
os: Option<&Os>,
arch: Option<&Arch>,
-) -> Result<Option<Config>, ProxmoxOciError> {
+) -> Result<Option<Config>, ParseAndExtractTarError> {
let (oci_tar_image, image_manifest, image_config) = parse_image(oci_tar_path, os, arch)?;
extract_image_rootfs(oci_tar_image, &image_manifest, &image_config, rootfs_path)?;
diff --git a/proxmox-oci/tests/helper/mod.rs b/proxmox-oci/tests/helper/mod.rs
index 56e51fda..bd541c9c 100644
--- a/proxmox-oci/tests/helper/mod.rs
+++ b/proxmox-oci/tests/helper/mod.rs
@@ -5,7 +5,7 @@ use std::path::{Path, PathBuf};
use anyhow::Result;
-use proxmox_oci::{Arch, Config, Os, ProxmoxOciError, parse_and_extract_tar_image};
+use proxmox_oci::{Arch, Config, Os, ParseAndExtractTarError, parse_and_extract_tar_image};
use proxmox_sys::fs::make_tmp_dir;
mod oci_tar_image_builder;
@@ -70,7 +70,7 @@ impl TestImage {
})
}
- pub fn extract(&self) -> Result<Option<Config>, ProxmoxOciError> {
+ pub fn extract(&self) -> Result<Option<Config>, ParseAndExtractTarError> {
parse_and_extract_tar_image(
&self.path,
&self.extract_dir,
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 17/26] oci: add a public pull_image_as_tar function
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (15 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 16/26] oci: rename ProxmoxOciError to ParseAndExtractTarError Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 18/26] pve-rs: oci: expose pull_image_as_tar Filip Schauer
` (8 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/Cargo.toml | 1 +
proxmox-oci/debian/control | 2 +
proxmox-oci/src/lib.rs | 111 +++++++++++++++++++++++++++++++++++++
3 files changed, 114 insertions(+)
diff --git a/proxmox-oci/Cargo.toml b/proxmox-oci/Cargo.toml
index 45248658..87754e50 100644
--- a/proxmox-oci/Cargo.toml
+++ b/proxmox-oci/Cargo.toml
@@ -27,6 +27,7 @@ zstd.workspace = true
proxmox-base64.workspace = true
proxmox-http = { workspace = true, features = [ "client-sync" ] }
proxmox-io.workspace = true
+proxmox-lang.workspace = true
[dev-dependencies]
anyhow.workspace = true
diff --git a/proxmox-oci/debian/control b/proxmox-oci/debian/control
index 9ec73059..62dc2cd2 100644
--- a/proxmox-oci/debian/control
+++ b/proxmox-oci/debian/control
@@ -14,6 +14,7 @@ Build-Depends-Arch: cargo:native <!nocheck>,
librust-proxmox-http-1+client-sync-dev (>= 1.0.5-~~) <!nocheck>,
librust-proxmox-http-1+default-dev (>= 1.0.5-~~) <!nocheck>,
librust-proxmox-io-1+default-dev (>= 1.2.1-~~) <!nocheck>,
+ librust-proxmox-lang-1+default-dev (>= 1.5-~~) <!nocheck>,
librust-serde-1+default-dev <!nocheck>,
librust-serde-json-1+default-dev <!nocheck>,
librust-sha2-0.10+default-dev <!nocheck>,
@@ -41,6 +42,7 @@ Depends:
librust-proxmox-http-1+client-sync-dev (>= 1.0.5-~~),
librust-proxmox-http-1+default-dev (>= 1.0.5-~~),
librust-proxmox-io-1+default-dev (>= 1.2.1-~~),
+ librust-proxmox-lang-1+default-dev (>= 1.5-~~),
librust-serde-1+default-dev,
librust-serde-json-1+default-dev,
librust-sha2-0.10+default-dev,
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index 672d81e0..542c9cee 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -1,9 +1,15 @@
+use std::fs::{File, remove_file};
+use std::io;
use std::path::Path;
+use oci_spec::OciSpecError;
pub use oci_spec::distribution::Reference;
pub use oci_spec::image::{Arch, Config, Os};
+use oci_spec::image::{Descriptor, ImageIndexBuilder, MediaType, OciLayoutBuilder};
use thiserror::Error;
+use proxmox_lang::try_block;
+
pub mod client;
use client::{Client, ClientConfig, OciClientError, RegistryAuth};
@@ -69,3 +75,108 @@ pub fn list_repository_tags(
let tag_list = client.list_tags(reference, None, None)?;
Ok(tag_list.tags().to_vec())
}
+
+#[derive(Debug, Error)]
+pub enum PullTarError {
+ #[error(transparent)]
+ Client(#[from] OciClientError),
+ #[error(transparent)]
+ Io(#[from] io::Error),
+ #[error(transparent)]
+ OciSpec(#[from] OciSpecError),
+}
+
+/// Pull an OCI image from a registry and save it as a tar archive.
+///
+/// This removes the tar archive on error.
+///
+/// # Arguments
+///
+/// * `reference` - OCI reference specifying the image repository and tag/digest to pull
+/// * `auth` - Authentication credentials/method to access the registry
+/// * `tar_path` - Destination file path where the tar archive will be written to
+pub fn pull_image_as_tar(
+ reference: &Reference,
+ auth: RegistryAuth,
+ tar_path: impl AsRef<Path>,
+) -> Result<(), PullTarError> {
+ let mut client = Client::new(ClientConfig::default(), auth);
+ println!("Pulling image manifest and image config");
+ let meta = client.pull_image_manifest_and_config(reference, None, None)?;
+
+ let layout = OciLayoutBuilder::default()
+ .image_layout_version("1.0.0")
+ .build()?;
+ let manifest_descriptor = Descriptor::new(
+ MediaType::ImageManifest,
+ meta.manifest.raw.len() as u64,
+ meta.manifest.digest.clone(),
+ );
+ let index = ImageIndexBuilder::default()
+ .schema_version(2u32)
+ .manifests([manifest_descriptor])
+ .build()?;
+
+ let oci_tar_file = File::create(&tar_path)?;
+ let mut tar = tar::Builder::new(oci_tar_file);
+
+ try_block!({
+ let mut layout_header = tar::Header::new_old();
+ let mut layout_entry = tar.append_writer(&mut layout_header, "oci-layout")?;
+ layout.to_writer(&mut layout_entry)?;
+ layout_entry.finish()?;
+
+ let mut index_header = tar::Header::new_old();
+ let mut index_entry = tar.append_writer(&mut index_header, "index.json")?;
+ index.to_writer(&mut index_entry)?;
+ index_entry.finish()?;
+
+ let mut manifest_header = tar::Header::new_old();
+ manifest_header.set_size(meta.manifest.raw.len() as u64);
+ manifest_header.set_cksum();
+ tar.append_data(
+ &mut manifest_header,
+ format!(
+ "blobs/{}/{}",
+ meta.manifest.digest.algorithm(),
+ meta.manifest.digest.digest()
+ ),
+ &meta.manifest.raw[..],
+ )?;
+
+ let mut config_header = tar::Header::new_old();
+ config_header.set_size(meta.config.raw.len() as u64);
+ config_header.set_cksum();
+ tar.append_data(
+ &mut config_header,
+ format!(
+ "blobs/{}/{}",
+ meta.config.digest.algorithm(),
+ meta.config.digest.digest()
+ ),
+ &meta.config.raw[..],
+ )?;
+
+ for layer in meta.manifest.manifest.layers() {
+ let layer_digest = layer.digest();
+ println!("Pulling blob {layer_digest}");
+ let mut layer_header = tar::Header::new_old();
+ let mut layer_entry = tar.append_writer(
+ &mut layer_header,
+ format!(
+ "blobs/{}/{}",
+ layer_digest.algorithm(),
+ layer_digest.digest()
+ ),
+ )?;
+ client.pull_blob(reference, layer, &mut layer_entry)?;
+ layer_entry.finish()?;
+ }
+
+ Ok(tar.finish()?)
+ })
+ .inspect_err(|_| {
+ drop(tar);
+ let _ = remove_file(tar_path);
+ })
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox-perl-rs 18/26] pve-rs: oci: expose pull_image_as_tar
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (16 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 17/26] oci: add a public pull_image_as_tar function Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH storage 19/26] use proxmox-oci instead of skopeo for pulling from OCI registries Filip Schauer
` (7 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
pve-rs/src/bindings/oci.rs | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/pve-rs/src/bindings/oci.rs b/pve-rs/src/bindings/oci.rs
index e26f336..bbe6b34 100644
--- a/pve-rs/src/bindings/oci.rs
+++ b/pve-rs/src/bindings/oci.rs
@@ -54,4 +54,23 @@ pub mod pve_rs_oci {
let reference = Reference::from_str(reference)?;
Ok(proxmox_oci::list_repository_tags(&reference, auth)?)
}
+
+ /// Pull an OCI image from a registry and save it as a tar archive.
+ ///
+ /// This removes the tar archive on error.
+ ///
+ /// # Arguments
+ ///
+ /// * `reference` - OCI reference specifying the image repository and tag/digest to pull
+ /// * `auth` - Authentication credentials/method to access the registry
+ /// * `tar_path` - Destination file path where the tar archive will be written to
+ #[export]
+ pub fn pull_image_as_tar(
+ reference: &str,
+ auth: RegistryAuth,
+ tar_path: &str,
+ ) -> Result<(), Error> {
+ let reference = Reference::from_str(reference)?;
+ Ok(proxmox_oci::pull_image_as_tar(&reference, auth, tar_path)?)
+ }
}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH storage 19/26] use proxmox-oci instead of skopeo for pulling from OCI registries
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (17 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 18/26] pve-rs: oci: expose pull_image_as_tar Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH storage 20/26] api: oci image pull: fix temp file cleanup on task stop Filip Schauer
` (6 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
debian/control | 1 -
src/PVE/API2/Storage/Status.pm | 8 +-------
2 files changed, 1 insertion(+), 8 deletions(-)
diff --git a/debian/control b/debian/control
index 3a86607..7223458 100644
--- a/debian/control
+++ b/debian/control
@@ -55,7 +55,6 @@ Depends: bzip2,
${misc:Depends},
${perl:Depends},
Recommends: pve-esxi-import-tools (>= 0.6.0),
- skopeo,
zfsutils-linux (>= 2.4.1-pve2),
zfs-zed,
Description: Proxmox VE storage management library
diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 57ab92b..8812d47 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -952,8 +952,6 @@ __PACKAGE__->register_method({
code => sub {
my ($param) = @_;
- die "Install 'skopeo' to pull OCI images from registries.\n" if (!-f '/usr/bin/skopeo');
-
my $rpcenv = PVE::RPCEnvironment::get();
my $user = $rpcenv->get_user();
@@ -986,11 +984,7 @@ __PACKAGE__->register_method({
};
my $worker = sub {
- PVE::Tools::run_command(
- [
- "skopeo", "copy", "docker://$reference", "oci-archive:$path/$tmp_filename",
- ],
- );
+ PVE::RS::OCI::pull_image_as_tar($reference, "Anonymous", "$path/$tmp_filename");
rename("$path/$tmp_filename", "$path/$filename")
or die "unable to rename temporary file: $!\n";
};
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH storage 20/26] api: oci image pull: fix temp file cleanup on task stop
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (18 preceding siblings ...)
2026-11-12 14:10 ` [PATCH storage 19/26] use proxmox-oci instead of skopeo for pulling from OCI registries Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 21/26] oci: add a public pull_image_to_oci_storage function Filip Schauer
` (5 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
The $SIG{INT} handler was installed in the parent process before
fork_worker() forks the actual worker child. fork_worker() overwrites
$SIG{INT}/TERM/QUIT in the child with its own handler before invoking
the worker sub, so the cleanup handler never ran and interrupted pulls
left the temporary file behind.
Move the handler into the worker and also catch SIGTERM, since that is
the signal actually sent when a task is stopped.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/API2/Storage/Status.pm | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 8812d47..332e06d 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -976,14 +976,14 @@ __PACKAGE__->register_method({
die "refusing to override existing file '$filename'\n" if -f "$path/$filename";
- local $SIG{INT} = sub {
- unlink "$path/$tmp_filename"
- or $!{ENOENT}
- or warn "could not cleanup temporary file: $!";
- die "got interrupted by signal\n";
- };
-
my $worker = sub {
+ local $SIG{INT} = local $SIG{QUIT} = local $SIG{TERM} = sub {
+ unlink "$path/$tmp_filename"
+ or $!{ENOENT}
+ or warn "could not cleanup temporary file: $!";
+ die "got interrupted by signal\n";
+ };
+
PVE::RS::OCI::pull_image_as_tar($reference, "Anonymous", "$path/$tmp_filename");
rename("$path/$tmp_filename", "$path/$filename")
or die "unable to rename temporary file: $!\n";
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 21/26] oci: add a public pull_image_to_oci_storage function
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (19 preceding siblings ...)
2026-11-12 14:10 ` [PATCH storage 20/26] api: oci image pull: fix temp file cleanup on task stop Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 22/26] pve-rs: oci: expose pull_image_to_oci_storage Filip Schauer
` (4 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Also bump the MSRV (Minimum Supported Rust Version) to 1.91 because of
Path::with_added_extension.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/Cargo.toml | 5 +-
proxmox-oci/debian/control | 6 +-
| 50 +++++++++++-
proxmox-oci/src/lib.rs | 159 ++++++++++++++++++++++++++++++++++++-
4 files changed, 210 insertions(+), 10 deletions(-)
diff --git a/proxmox-oci/Cargo.toml b/proxmox-oci/Cargo.toml
index 87754e50..6c5f9a7f 100644
--- a/proxmox-oci/Cargo.toml
+++ b/proxmox-oci/Cargo.toml
@@ -9,12 +9,13 @@ exclude.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
-rust-version.workspace = true
+rust-version = "1.91"
[dependencies]
flate2.workspace = true
http.workspace = true
http-auth = { version = "0.1", default-features = false }
+nix.workspace = true
oci-spec = "0.8.3"
serde.workspace = true
serde_json.workspace = true
@@ -28,8 +29,8 @@ proxmox-base64.workspace = true
proxmox-http = { workspace = true, features = [ "client-sync" ] }
proxmox-io.workspace = true
proxmox-lang.workspace = true
+proxmox-sys.workspace = true
[dev-dependencies]
anyhow.workspace = true
-proxmox-sys.workspace = true
nix = { workspace = true, features = ["user"] }
diff --git a/proxmox-oci/debian/control b/proxmox-oci/debian/control
index 62dc2cd2..f38a6e35 100644
--- a/proxmox-oci/debian/control
+++ b/proxmox-oci/debian/control
@@ -4,17 +4,19 @@ Priority: optional
Build-Depends: debhelper-compat (= 13),
dh-sequence-cargo
Build-Depends-Arch: cargo:native <!nocheck>,
- rustc:native (>= 1.88) <!nocheck>,
+ rustc:native (>= 1.91) <!nocheck>,
libstd-rust-dev <!nocheck>,
librust-flate2-1+default-dev <!nocheck>,
librust-http-1+default-dev <!nocheck>,
librust-http-auth-0.1-dev <!nocheck>,
+ librust-nix-0.29+default-dev <!nocheck>,
librust-oci-spec-0.8+default-dev (>= 0.8.3-~~) <!nocheck>,
librust-proxmox-base64-1+default-dev <!nocheck>,
librust-proxmox-http-1+client-sync-dev (>= 1.0.5-~~) <!nocheck>,
librust-proxmox-http-1+default-dev (>= 1.0.5-~~) <!nocheck>,
librust-proxmox-io-1+default-dev (>= 1.2.1-~~) <!nocheck>,
librust-proxmox-lang-1+default-dev (>= 1.5-~~) <!nocheck>,
+ librust-proxmox-sys-1+default-dev (>= 1.0.1-~~) <!nocheck>,
librust-serde-1+default-dev <!nocheck>,
librust-serde-json-1+default-dev <!nocheck>,
librust-sha2-0.10+default-dev <!nocheck>,
@@ -37,12 +39,14 @@ Depends:
librust-flate2-1+default-dev,
librust-http-1+default-dev,
librust-http-auth-0.1-dev,
+ librust-nix-0.29+default-dev,
librust-oci-spec-0.8+default-dev (>= 0.8.3-~~),
librust-proxmox-base64-1+default-dev,
librust-proxmox-http-1+client-sync-dev (>= 1.0.5-~~),
librust-proxmox-http-1+default-dev (>= 1.0.5-~~),
librust-proxmox-io-1+default-dev (>= 1.2.1-~~),
librust-proxmox-lang-1+default-dev (>= 1.5-~~),
+ librust-proxmox-sys-1+default-dev (>= 1.0.1-~~),
librust-serde-1+default-dev,
librust-serde-json-1+default-dev,
librust-sha2-0.10+default-dev,
--git a/proxmox-oci/src/extract.rs b/proxmox-oci/src/extract.rs
index b158c3be..979c93eb 100644
--- a/proxmox-oci/src/extract.rs
+++ b/proxmox-oci/src/extract.rs
@@ -1,12 +1,14 @@
-use std::ffi::OsStr;
-use std::fs::{File, read_dir, remove_dir_all, remove_file};
+use std::ffi::{CString, OsStr};
+use std::fs::{File, create_dir_all, read_dir, remove_dir_all, remove_file};
use std::io::{self, ErrorKind, Read, Seek, SeekFrom};
+use std::os::fd::AsRawFd;
use std::os::unix::ffi::OsStrExt;
use std::path::{Path, PathBuf};
use std::str::FromStr;
use std::time::{Duration, UNIX_EPOCH};
use flate2::read::GzDecoder;
+use nix::sys::stat::{Mode, SFlag, makedev, mknod};
use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType};
use tar::{Archive, EntryType};
use thiserror::Error;
@@ -81,7 +83,7 @@ pub(crate) fn extract_image_rootfs<R: Read + Seek, P: AsRef<Path>>(
)?;
layer_file.seek(SeekFrom::Start(0))?;
- extract_archive(&mut decode_fn(&mut layer_file)?, &target_path)?;
+ extract_layer(decode_fn(&mut layer_file)?, &target_path, false)?;
oci_tar_image = layer_file.into_oci_tar_image();
}
@@ -216,7 +218,11 @@ fn apply_whiteout(target_path: &Path, path: &Path) -> io::Result<()> {
Ok(())
}
-fn extract_archive<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()> {
+pub(crate) fn extract_layer<R: Read>(
+ reader: R,
+ target_path: &Path,
+ create_whiteouts: bool,
+) -> io::Result<()> {
let mut archive = Archive::new(reader);
archive.set_preserve_ownerships(true);
archive.set_preserve_permissions(true);
@@ -235,6 +241,42 @@ fn extract_archive<R: Read>(reader: &mut R, target_path: &Path) -> io::Result<()
directories.push(file);
continue;
} else if file.header().entry_type() == EntryType::Regular && is_whiteout(&filepath) {
+ // Handle whiteout files
+ if create_whiteouts && let Some(filename) = filepath.file_name() {
+ if filename == OPAQUE_WHITEOUT_NAME {
+ if let Some(parent) = filepath.parent() {
+ let whiteout_abs_path = target_path.join(parent);
+
+ // Create opaque whiteout
+ create_dir_all(&whiteout_abs_path)?;
+ let dir = File::open(whiteout_abs_path)?;
+ let fd = dir.as_raw_fd();
+ proxmox_sys::fs::xattr::fsetxattr(
+ fd,
+ &CString::new("trusted.overlay.opaque").unwrap(),
+ b"y",
+ )?;
+ }
+ } else if let Some(filename_stripped) =
+ // TODO: Simplify this once OsStr::strip_prefix is implemented
+ filename.as_encoded_bytes().strip_prefix(WHITEOUT_PREFIX)
+ {
+ let whiteout_path = match filename_stripped {
+ b"." => match filepath.parent() {
+ Some(p) if p.parent().is_some() => p,
+ _ => continue, // Prevent whiteout of root directory
+ },
+ b".." => continue, // Prevent whiteout of grandparent directory
+ fname => &filepath.with_file_name(OsStr::from_bytes(fname)),
+ };
+ let whiteout_abs_path = target_path.join(whiteout_path);
+
+ // Create whiteout node
+ create_dir_all(whiteout_abs_path.parent().unwrap())?;
+ let dev = makedev(0, 0);
+ mknod(&whiteout_abs_path, SFlag::S_IFCHR, Mode::S_IRUSR, dev)?;
+ }
+ }
continue;
}
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index 542c9cee..d7680cce 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -1,7 +1,9 @@
-use std::fs::{File, remove_file};
-use std::io;
+use std::fs::{File, create_dir_all, remove_dir_all, remove_file, rename};
+use std::io::{self, Read};
use std::path::Path;
+use std::str::FromStr;
+use flate2::read::GzDecoder;
use oci_spec::OciSpecError;
pub use oci_spec::distribution::Reference;
pub use oci_spec::image::{Arch, Config, Os};
@@ -9,15 +11,17 @@ use oci_spec::image::{Descriptor, ImageIndexBuilder, MediaType, OciLayoutBuilder
use thiserror::Error;
use proxmox_lang::try_block;
+use proxmox_sys::fs::{CreateOptions, replace_file};
pub mod client;
use client::{Client, ClientConfig, OciClientError, RegistryAuth};
mod digest;
+use digest::{DigestError, VerifyingReader};
mod extract;
pub use extract::ExtractError;
-use extract::extract_image_rootfs;
+use extract::{extract_image_rootfs, extract_layer};
mod parse;
pub use parse::ParseError;
@@ -180,3 +184,152 @@ pub fn pull_image_as_tar(
let _ = remove_file(tar_path);
})
}
+
+#[derive(Debug, Error)]
+pub enum PullToStorageError {
+ #[error(transparent)]
+ Client(#[from] OciClientError),
+ #[error(transparent)]
+ Digest(#[from] DigestError),
+ #[error(transparent)]
+ Io(#[from] io::Error),
+ #[error(transparent)]
+ OciSpec(#[from] OciSpecError),
+}
+
+/// Pull an OCI image from a registry and save it to a storage with the "oci" content type.
+///
+/// This cleans up unfinished layers on error. However it does not remove layers that were
+/// already successfully written, even if the download fails at another layer.
+///
+/// # Arguments
+///
+/// * `reference` - OCI reference specifying the image repository and tag/digest to pull
+/// * `auth` - Authentication credentials/method to access the registry
+/// * `oci_storage_path` - Path to the target "oci" storage directory
+///
+/// # Returns
+///
+/// On success, this returns the Image ID of the pulled image in the format `<algorithm>-<hex>`.
+pub fn pull_image_to_oci_storage(
+ reference: &Reference,
+ auth: RegistryAuth,
+ oci_storage_path: impl AsRef<Path>,
+) -> Result<String, PullToStorageError> {
+ let mut client = Client::new(ClientConfig::default(), auth);
+ println!("Pulling image manifest and image config");
+ let meta = client.pull_image_manifest_and_config(reference, None, None)?;
+ let digested_manifest = meta.manifest;
+ let digested_config = meta.config;
+ let manifest = digested_manifest.manifest;
+ let config = digested_config.config;
+
+ let layers_path = oci_storage_path.as_ref().join("layers");
+ let layers_by_blob_id_path = layers_path.join("by-blob-id");
+
+ for (layer, diff_id_str) in manifest.layers().iter().zip(config.rootfs().diff_ids()) {
+ let blob_digest = layer.digest();
+ let diff_id = oci_spec::image::Digest::from_str(diff_id_str)?;
+ let layers_by_blob_id_blob_digest_path = layers_by_blob_id_path.join(format!(
+ "{}-{}",
+ blob_digest.algorithm(),
+ blob_digest.digest()
+ ));
+ let layers_diff_id_path =
+ layers_path.join(format!("{}-{}", diff_id.algorithm(), diff_id.digest()));
+
+ if layers_diff_id_path.exists() {
+ println!("Skipping download of blob {blob_digest}, already present");
+ } else {
+ println!("Pulling blob {blob_digest}");
+ let layer_reader = client.pull_blob_reader(reference, layer)?;
+ let mut uncompressed_blob = match layer.media_type() {
+ MediaType::ImageLayer | MediaType::ImageLayerNonDistributable => {
+ if blob_digest != &diff_id {
+ return Err(PullToStorageError::Digest(DigestError::Mismatch {
+ expected: blob_digest.clone(),
+ actual: diff_id,
+ }));
+ }
+ layer_reader
+ }
+ MediaType::ImageLayerGzip | MediaType::ImageLayerNonDistributableGzip => {
+ VerifyingReader::<Box<dyn Read>>::new(
+ Box::new(GzDecoder::new(layer_reader)),
+ diff_id,
+ )?
+ }
+ MediaType::ImageLayerZstd | MediaType::ImageLayerNonDistributableZstd => {
+ VerifyingReader::<Box<dyn Read>>::new(
+ // https://github.com/gyscos/zstd-rs/issues/281
+ Box::new(zstd::Decoder::new(layer_reader).expect("infallible")),
+ diff_id,
+ )?
+ }
+ // Skip any other non-ImageLayer related media types.
+ // Match explicitly to avoid missing new image layer types when oci-spec updates.
+ MediaType::Descriptor
+ | MediaType::LayoutHeader
+ | MediaType::ImageManifest
+ | MediaType::ImageIndex
+ | MediaType::ImageConfig
+ | MediaType::ArtifactManifest
+ | MediaType::EmptyJSON
+ | MediaType::Other(_) => continue,
+ };
+
+ let temp_layers_diff_id_path = layers_diff_id_path.with_added_extension("tmp");
+ if temp_layers_diff_id_path.exists() {
+ // Clean up a potential dangling .tmp directory first.
+ remove_dir_all(&temp_layers_diff_id_path)?;
+ }
+ create_dir_all(&temp_layers_diff_id_path)?;
+
+ try_block!({
+ extract_layer(&mut uncompressed_blob, &temp_layers_diff_id_path, true)?;
+
+ // Force an EOF, so VerifyingReader actually verifies.
+ io::copy(&mut uncompressed_blob, &mut io::sink())?;
+
+ rename(&temp_layers_diff_id_path, layers_diff_id_path)?;
+ Ok(())
+ })
+ .inspect_err(|_: &io::Error| {
+ // Do not leave a dangling .tmp directory behind.
+ let _ = remove_dir_all(&temp_layers_diff_id_path);
+ })?;
+ }
+
+ if !layers_by_blob_id_blob_digest_path.exists() {
+ println!("Mapping blob {blob_digest} to layer DiffID {diff_id_str}");
+ create_dir_all(&layers_by_blob_id_path)?;
+ replace_file(
+ &layers_by_blob_id_blob_digest_path,
+ diff_id_str.as_bytes(),
+ CreateOptions::new(),
+ true,
+ )
+ .map_err(io::Error::other)?;
+ }
+ }
+
+ println!(
+ "Writing image config for ImageID {}",
+ digested_config.digest
+ );
+ let image_id = format!(
+ "{}-{}",
+ digested_config.digest.algorithm(),
+ digested_config.digest.digest()
+ );
+ let image_config_path = oci_storage_path.as_ref().join(&image_id);
+ replace_file(
+ image_config_path,
+ &digested_config.raw,
+ CreateOptions::new(),
+ true,
+ )
+ .map_err(io::Error::other)?;
+
+ Ok(image_id)
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox-perl-rs 22/26] pve-rs: oci: expose pull_image_to_oci_storage
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (20 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 21/26] oci: add a public pull_image_to_oci_storage function Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 23/26] fix #7058: oci: add compatibility for Docker Image Manifest V2 Filip Schauer
` (3 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
pve-rs/src/bindings/oci.rs | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/pve-rs/src/bindings/oci.rs b/pve-rs/src/bindings/oci.rs
index bbe6b34..aec7985 100644
--- a/pve-rs/src/bindings/oci.rs
+++ b/pve-rs/src/bindings/oci.rs
@@ -73,4 +73,32 @@ pub mod pve_rs_oci {
let reference = Reference::from_str(reference)?;
Ok(proxmox_oci::pull_image_as_tar(&reference, auth, tar_path)?)
}
+
+ /// Pull an OCI image from a registry and save it to a storage with the "oci" content type.
+ ///
+ /// This cleans up unfinished layers on error. However it does not remove layers that were
+ /// already successfully written, even if the download fails at another layer.
+ ///
+ /// # Arguments
+ ///
+ /// * `reference` - OCI reference specifying the image repository and tag/digest to pull
+ /// * `auth` - Authentication credentials/method to access the registry
+ /// * `oci_storage_path` - Path to the target "oci" storage directory
+ ///
+ /// # Returns
+ ///
+ /// On success, this returns the Image ID of the pulled image in the format `<algorithm>-<hex>`.
+ #[export]
+ pub fn pull_image_to_oci_storage(
+ reference: &str,
+ auth: RegistryAuth,
+ oci_storage_path: &str,
+ ) -> Result<String, Error> {
+ let reference = Reference::from_str(reference)?;
+ Ok(proxmox_oci::pull_image_to_oci_storage(
+ &reference,
+ auth,
+ oci_storage_path,
+ )?)
+ }
}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 23/26] fix #7058: oci: add compatibility for Docker Image Manifest V2
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (21 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 22/26] pve-rs: oci: expose pull_image_to_oci_storage Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 24/26] oci: include media type in ParseError::WrongMediaType Filip Schauer
` (2 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Many container registries and build tools use Docker Image Manifest V2
media types instead of pure OCI media types. This caused pulling or
extracting such images to fail due to unexpected media types.
Add compatibility by accepting the corresponding Docker Image Manifest
V2 media types alongside the OCI media types.
To test this on an OCI TAR archive, one can use Docker to generate an
OCI image with Docker media types:
cat << 'EOF' > Dockerfile
FROM alpine:3.18
CMD ["sh"]
EOF
docker buildx build \
--output type=oci,dest=out.tar,oci-mediatypes=false .
To test pulling an OCI image with a Docker Manifest List from a
registry, one can pull `grafana/grafana:13.2-slim` from Docker Hub.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/client.rs | 16 +++++++++++++---
| 13 +++++++++++--
proxmox-oci/src/lib.rs | 14 ++++++++++++--
proxmox-oci/src/oci_tar_image.rs | 5 +++--
proxmox-oci/src/parse.rs | 7 +++++--
5 files changed, 44 insertions(+), 11 deletions(-)
diff --git a/proxmox-oci/src/client.rs b/proxmox-oci/src/client.rs
index c6d15cbc..5d5beb8c 100644
--- a/proxmox-oci/src/client.rs
+++ b/proxmox-oci/src/client.rs
@@ -11,6 +11,7 @@ use oci_spec::OciSpecError;
use oci_spec::distribution::{ErrorResponse, Reference, TagList};
use oci_spec::image::{
Arch, Descriptor, Digest, ImageConfiguration, ImageIndex, ImageManifest, MediaType, Os,
+ ToDockerV2S2,
};
use proxmox_http::client::sync;
use proxmox_http::{HttpClient, HttpOptions, ProxyConfig};
@@ -405,10 +406,14 @@ impl Client {
&[
MediaType::ImageManifest.as_ref(),
MediaType::ImageIndex.as_ref(),
+ MediaType::ImageManifest.to_docker_v2s2().unwrap(),
+ MediaType::ImageIndex.to_docker_v2s2().unwrap(),
],
)?;
- if matches!(media_type, MediaType::ImageIndex) {
+ if media_type == MediaType::ImageIndex
+ || media_type.as_ref() == MediaType::ImageIndex.to_docker_v2s2().unwrap()
+ {
let index = ImageIndex::from_reader(Cursor::new(&body))?;
let manifest_digest = index
@@ -423,8 +428,13 @@ impl Client {
.ok_or(OciClientError::NoMatchingManifest)?;
let manifest_reference = image.clone_with_digest(manifest_digest.to_string());
- let (manifest_raw, manifest_digest, _) =
- self.pull_manifest_raw(&manifest_reference, &[MediaType::ImageManifest.as_ref()])?;
+ let (manifest_raw, manifest_digest, _) = self.pull_manifest_raw(
+ &manifest_reference,
+ &[
+ MediaType::ImageManifest.as_ref(),
+ MediaType::ImageManifest.to_docker_v2s2().unwrap(),
+ ],
+ )?;
let manifest = ImageManifest::from_reader(Cursor::new(&manifest_raw))?;
Ok(DigestedImageManifest {
--git a/proxmox-oci/src/extract.rs b/proxmox-oci/src/extract.rs
index 979c93eb..0bb272c3 100644
--- a/proxmox-oci/src/extract.rs
+++ b/proxmox-oci/src/extract.rs
@@ -9,7 +9,7 @@ use std::time::{Duration, UNIX_EPOCH};
use flate2::read::GzDecoder;
use nix::sys::stat::{Mode, SFlag, makedev, mknod};
-use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType};
+use oci_spec::image::{ImageConfiguration, ImageManifest, MediaType, ToDockerV2S2};
use tar::{Archive, EntryType};
use thiserror::Error;
@@ -52,8 +52,17 @@ pub(crate) fn extract_image_rootfs<R: Read + Seek, P: AsRef<Path>>(
ExtractError::MissingLayerFile(layer_descriptor.digest().clone()),
)?;
+ let media_type = match layer_descriptor.media_type() {
+ MediaType::Other(other)
+ if other == MediaType::ImageLayerGzip.to_docker_v2s2().unwrap() =>
+ {
+ &MediaType::ImageLayerGzip
+ }
+ other => other,
+ };
+
type DecodeFn<T> = Box<dyn for<'a> Fn(&'a mut T) -> io::Result<Box<dyn Read + 'a>>>;
- let decode_fn: DecodeFn<OciTarImageBlob<R>> = match layer_descriptor.media_type() {
+ let decode_fn: DecodeFn<OciTarImageBlob<R>> = match media_type {
MediaType::ImageLayer | MediaType::ImageLayerNonDistributable => {
Box::new(|file| Ok(Box::new(file)))
}
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index d7680cce..cd798fe3 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -7,7 +7,7 @@ use flate2::read::GzDecoder;
use oci_spec::OciSpecError;
pub use oci_spec::distribution::Reference;
pub use oci_spec::image::{Arch, Config, Os};
-use oci_spec::image::{Descriptor, ImageIndexBuilder, MediaType, OciLayoutBuilder};
+use oci_spec::image::{Descriptor, ImageIndexBuilder, MediaType, OciLayoutBuilder, ToDockerV2S2};
use thiserror::Error;
use proxmox_lang::try_block;
@@ -243,7 +243,17 @@ pub fn pull_image_to_oci_storage(
} else {
println!("Pulling blob {blob_digest}");
let layer_reader = client.pull_blob_reader(reference, layer)?;
- let mut uncompressed_blob = match layer.media_type() {
+
+ let media_type = match layer.media_type() {
+ MediaType::Other(other)
+ if other == MediaType::ImageLayerGzip.to_docker_v2s2().unwrap() =>
+ {
+ &MediaType::ImageLayerGzip
+ }
+ other => other,
+ };
+
+ let mut uncompressed_blob = match media_type {
MediaType::ImageLayer | MediaType::ImageLayerNonDistributable => {
if blob_digest != &diff_id {
return Err(PullToStorageError::Digest(DigestError::Mismatch {
diff --git a/proxmox-oci/src/oci_tar_image.rs b/proxmox-oci/src/oci_tar_image.rs
index f51556e9..cca9da72 100644
--- a/proxmox-oci/src/oci_tar_image.rs
+++ b/proxmox-oci/src/oci_tar_image.rs
@@ -4,7 +4,7 @@ use std::ops::Range;
use std::path::{Path, PathBuf};
use oci_spec::OciSpecError;
-use oci_spec::image::{Arch, Digest, ImageIndex, ImageManifest, MediaType, Os};
+use oci_spec::image::{Arch, Digest, ImageIndex, ImageManifest, MediaType, Os, ToDockerV2S2};
use tar::Archive;
use proxmox_io::RangeReader;
@@ -77,7 +77,8 @@ impl<R: Read + Seek> OciTarImage<R> {
architecture: Option<&Arch>,
) -> Option<oci_spec::Result<ImageManifest>> {
let digest = match self.image_index.manifests().iter().find(|d| {
- d.media_type() == &MediaType::ImageManifest
+ (d.media_type() == &MediaType::ImageManifest
+ || d.media_type().as_ref() == MediaType::ImageManifest.to_docker_v2s2().unwrap())
&& d.platform().as_ref().is_none_or(|p| {
architecture.is_none_or(|a| p.architecture() == a)
&& os.is_none_or(|os| p.os() == os)
diff --git a/proxmox-oci/src/parse.rs b/proxmox-oci/src/parse.rs
index af669d38..1dc4eea2 100644
--- a/proxmox-oci/src/parse.rs
+++ b/proxmox-oci/src/parse.rs
@@ -2,7 +2,7 @@ use std::fs::File;
use std::path::Path;
use oci_spec::OciSpecError;
-use oci_spec::image::{Arch, ImageConfiguration, ImageManifest, MediaType, Os};
+use oci_spec::image::{Arch, ImageConfiguration, ImageManifest, MediaType, Os, ToDockerV2S2};
use thiserror::Error;
use crate::oci_tar_image::OciTarImage;
@@ -35,7 +35,10 @@ pub(crate) fn parse_image<P: AsRef<Path>>(
let image_config_descriptor = image_manifest.config();
- if image_config_descriptor.media_type() != &MediaType::ImageConfig {
+ if image_config_descriptor.media_type() != &MediaType::ImageConfig
+ && image_config_descriptor.media_type().as_ref()
+ != MediaType::ImageConfig.to_docker_v2s2().unwrap()
+ {
return Err(ParseError::WrongMediaType);
}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 24/26] oci: include media type in ParseError::WrongMediaType
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (22 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 23/26] fix #7058: oci: add compatibility for Docker Image Manifest V2 Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox 25/26] oci: add public functions for removing OCI images Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 26/26] pve-rs: oci: expose " Filip Schauer
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/parse.rs | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/proxmox-oci/src/parse.rs b/proxmox-oci/src/parse.rs
index 1dc4eea2..34e82aa2 100644
--- a/proxmox-oci/src/parse.rs
+++ b/proxmox-oci/src/parse.rs
@@ -11,8 +11,8 @@ use crate::oci_tar_image::OciTarImage;
pub enum ParseError {
#[error(transparent)]
OciSpec(#[from] OciSpecError),
- #[error("Wrong media type")]
- WrongMediaType,
+ #[error("Image Config has wrong media type: {0}")]
+ WrongMediaType(String),
#[error(transparent)]
Io(#[from] std::io::Error),
#[error("No image manifest found in image index for the requested platform")]
@@ -39,7 +39,9 @@ pub(crate) fn parse_image<P: AsRef<Path>>(
&& image_config_descriptor.media_type().as_ref()
!= MediaType::ImageConfig.to_docker_v2s2().unwrap()
{
- return Err(ParseError::WrongMediaType);
+ return Err(ParseError::WrongMediaType(
+ image_config_descriptor.media_type().to_string(),
+ ));
}
let mut image_config_file = oci_tar_image
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox 25/26] oci: add public functions for removing OCI images
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (23 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 24/26] oci: include media type in ParseError::WrongMediaType Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
2026-11-12 14:10 ` [PATCH proxmox-perl-rs 26/26] pve-rs: oci: expose " Filip Schauer
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
proxmox-oci/src/lib.rs | 166 +++++++++++++++++++++++++++++++++++++----
1 file changed, 152 insertions(+), 14 deletions(-)
diff --git a/proxmox-oci/src/lib.rs b/proxmox-oci/src/lib.rs
index cd798fe3..2261feb4 100644
--- a/proxmox-oci/src/lib.rs
+++ b/proxmox-oci/src/lib.rs
@@ -1,4 +1,7 @@
-use std::fs::{File, create_dir_all, remove_dir_all, remove_file, rename};
+use std::collections::HashSet;
+use std::fs::{
+ File, create_dir_all, read_dir, read_to_string, remove_dir_all, remove_file, rename,
+};
use std::io::{self, Read};
use std::path::Path;
use std::str::FromStr;
@@ -7,7 +10,10 @@ use flate2::read::GzDecoder;
use oci_spec::OciSpecError;
pub use oci_spec::distribution::Reference;
pub use oci_spec::image::{Arch, Config, Os};
-use oci_spec::image::{Descriptor, ImageIndexBuilder, MediaType, OciLayoutBuilder, ToDockerV2S2};
+use oci_spec::image::{
+ Descriptor, Digest, ImageConfiguration, ImageIndexBuilder, MediaType, OciLayoutBuilder,
+ ToDockerV2S2,
+};
use thiserror::Error;
use proxmox_lang::try_block;
@@ -37,6 +43,21 @@ pub enum ParseAndExtractTarError {
ExtractError(#[from] ExtractError),
}
+fn digest_to_file_name(digest: &Digest) -> String {
+ format!("{}-{}", digest.algorithm(), digest.digest())
+}
+
+fn file_name_to_digest(name: &str) -> Option<Digest> {
+ let (algorithm, hex) = name.split_once('-')?;
+ format!("{algorithm}:{hex}").parse().ok()
+}
+
+fn is_valid_image_id(image_id: &str) -> bool {
+ image_id
+ .strip_prefix("sha256-")
+ .is_some_and(|hex| hex.len() == 64 && hex.bytes().all(|b| b.is_ascii_hexdigit()))
+}
+
/// Extract the rootfs of an OCI image tar and return its execution parameters.
///
/// # Arguments
@@ -230,13 +251,9 @@ pub fn pull_image_to_oci_storage(
for (layer, diff_id_str) in manifest.layers().iter().zip(config.rootfs().diff_ids()) {
let blob_digest = layer.digest();
let diff_id = oci_spec::image::Digest::from_str(diff_id_str)?;
- let layers_by_blob_id_blob_digest_path = layers_by_blob_id_path.join(format!(
- "{}-{}",
- blob_digest.algorithm(),
- blob_digest.digest()
- ));
- let layers_diff_id_path =
- layers_path.join(format!("{}-{}", diff_id.algorithm(), diff_id.digest()));
+ let layers_by_blob_id_blob_digest_path =
+ layers_by_blob_id_path.join(digest_to_file_name(blob_digest));
+ let layers_diff_id_path = layers_path.join(digest_to_file_name(&diff_id));
if layers_diff_id_path.exists() {
println!("Skipping download of blob {blob_digest}, already present");
@@ -327,11 +344,7 @@ pub fn pull_image_to_oci_storage(
"Writing image config for ImageID {}",
digested_config.digest
);
- let image_id = format!(
- "{}-{}",
- digested_config.digest.algorithm(),
- digested_config.digest.digest()
- );
+ let image_id = digest_to_file_name(&digested_config.digest);
let image_config_path = oci_storage_path.as_ref().join(&image_id);
replace_file(
image_config_path,
@@ -343,3 +356,128 @@ pub fn pull_image_to_oci_storage(
Ok(image_id)
}
+
+#[derive(Debug, Error)]
+pub enum RemoveFromStorageError {
+ #[error("ImageID has wrong format")]
+ InvalidImageId,
+ #[error(transparent)]
+ Io(#[from] io::Error),
+ #[error(transparent)]
+ OciSpec(#[from] OciSpecError),
+}
+
+/// Delete an OCI image configuration file from the storage and mark
+/// any layers that are no longer referenced by any remaining image for deletion.
+///
+/// # Arguments
+///
+/// * `image_id` - The ImageID i.e. the digest of the image configuration (e.g. `sha256-9f86d081...`)
+/// * `oci_storage_path` - Path to the root of the "oci" storage directory
+pub fn remove_image_from_oci_storage(
+ image_id: &str,
+ oci_storage_path: impl AsRef<Path>,
+) -> Result<(), RemoveFromStorageError> {
+ let oci_path = oci_storage_path.as_ref();
+ if !is_valid_image_id(image_id) {
+ return Err(RemoveFromStorageError::InvalidImageId);
+ }
+ let image_config_path = oci_path.join(image_id);
+ let image_config_diff_ids = ImageConfiguration::from_file(&image_config_path)?
+ .rootfs()
+ .diff_ids()
+ .iter()
+ .map(|s| s.parse())
+ .collect::<Result<HashSet<Digest>, _>>()?;
+
+ let mut still_referenced_diff_ids = HashSet::new();
+ for entry in read_dir(oci_path)? {
+ let entry = entry?;
+ if !entry.file_type()?.is_file() {
+ continue;
+ }
+
+ if !entry.file_name().to_str().is_some_and(is_valid_image_id) {
+ continue;
+ }
+
+ if entry.file_name() == image_id {
+ continue;
+ }
+
+ let config = ImageConfiguration::from_file(entry.path())?;
+ for diff_id in config.rootfs().diff_ids() {
+ still_referenced_diff_ids.insert(diff_id.parse()?);
+ }
+ }
+
+ println!("Removing image config for ImageID {image_id}");
+ remove_file(&image_config_path)?;
+
+ let layers_path = oci_path.join("layers");
+ for diff_id in image_config_diff_ids.difference(&still_referenced_diff_ids) {
+ let layer_path = layers_path.join(digest_to_file_name(diff_id));
+ println!("Marking layer with DiffID {diff_id} as unused");
+ let _ = rename(&layer_path, layer_path.with_added_extension("unused"));
+ }
+
+ Ok(())
+}
+
+#[derive(Debug, Error)]
+pub enum GarbageCollectStorageError {
+ #[error(transparent)]
+ Io(#[from] io::Error),
+ #[error(transparent)]
+ OciSpec(#[from] OciSpecError),
+}
+
+/// Garbage-collect unused layers and blob id to DiffID mappings from an OCI storage
+///
+/// # Arguments
+///
+/// * `oci_storage_path` - Path to the root of the "oci" storage directory
+pub fn garbage_collect_oci_storage(
+ oci_storage_path: impl AsRef<Path>,
+) -> Result<(), GarbageCollectStorageError> {
+ let oci_path = oci_storage_path.as_ref();
+ let layers_path = oci_path.join("layers");
+ let layers_by_blob_id_path = layers_path.join("by-blob-id");
+
+ let mut available_layers = HashSet::<Digest>::new();
+ let layers_direntries = match read_dir(layers_path) {
+ Ok(entries) => entries,
+ Err(err) if err.kind() == io::ErrorKind::NotFound => return Ok(()), // Nothing to do
+ Err(err) => return Err(err.into()),
+ };
+ for entry in layers_direntries.collect::<Vec<_>>() {
+ let entry = entry?;
+ if !entry.file_type()?.is_dir() {
+ continue;
+ }
+ if entry.file_name().as_encoded_bytes().ends_with(b".unused") {
+ let _ = remove_dir_all(entry.path());
+ } else if let Some(digest) = entry.file_name().to_str().and_then(file_name_to_digest) {
+ available_layers.insert(digest);
+ }
+ }
+
+ let blob_id_mapping_direntries = match read_dir(&layers_by_blob_id_path) {
+ Ok(entries) => entries,
+ Err(err) if err.kind() == io::ErrorKind::NotFound => return Ok(()), // Nothing to do
+ Err(err) => return Err(err.into()),
+ };
+ for entry in blob_id_mapping_direntries.collect::<Vec<_>>() {
+ let entry = entry?;
+ if !entry.file_type()?.is_file() {
+ continue;
+ }
+ let diff_id = Digest::from_str(&read_to_string(entry.path())?)?;
+ if !available_layers.contains(&diff_id) {
+ println!("Removing layer with DiffID {diff_id}");
+ let _ = remove_file(entry.path());
+ }
+ }
+
+ Ok(())
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH proxmox-perl-rs 26/26] pve-rs: oci: expose functions for removing OCI images
2026-11-12 14:10 [PATCH container/manager/proxmox{,-perl-rs}/storage 00/26] oci: add native registry client and prepare for OverlayFS Filip Schauer
` (24 preceding siblings ...)
2026-11-12 14:10 ` [PATCH proxmox 25/26] oci: add public functions for removing OCI images Filip Schauer
@ 2026-11-12 14:10 ` Filip Schauer
25 siblings, 0 replies; 27+ messages in thread
From: Filip Schauer @ 2026-11-12 14:10 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
pve-rs/src/bindings/oci.rs | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/pve-rs/src/bindings/oci.rs b/pve-rs/src/bindings/oci.rs
index aec7985..7c50910 100644
--- a/pve-rs/src/bindings/oci.rs
+++ b/pve-rs/src/bindings/oci.rs
@@ -101,4 +101,32 @@ pub mod pve_rs_oci {
oci_storage_path,
)?)
}
+
+ /// Delete an OCI image configuration file from the storage and mark
+ /// any layers that are no longer referenced by any remaining image for deletion.
+ ///
+ /// # Arguments
+ ///
+ /// * `image_id` - The ImageID i.e. the digest of the image configuration (e.g. `sha256-9f86d081...`)
+ /// * `oci_storage_path` - Path to the root of the "oci" storage directory
+ #[export]
+ pub fn remove_image_from_oci_storage(
+ image_id: &str,
+ oci_storage_path: &str,
+ ) -> Result<(), Error> {
+ Ok(proxmox_oci::remove_image_from_oci_storage(
+ image_id,
+ oci_storage_path,
+ )?)
+ }
+
+ /// Garbage-collect unused layers and blob id to DiffID mappings from an OCI storage
+ ///
+ /// # Arguments
+ ///
+ /// * `oci_storage_path` - Path to the root of the "oci" storage directory
+ #[export]
+ pub fn garbage_collect_oci_storage(oci_storage_path: &str) -> Result<(), Error> {
+ Ok(proxmox_oci::garbage_collect_oci_storage(oci_storage_path)?)
+ }
}
--
2.47.3
^ permalink raw reply related [flat|nested] 27+ messages in thread