[pve-devel] S3 Support

[pve-devel] S3 Support

[Tiziano Bacocco via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 4896 bytes --]

From: Tiziano Bacocco <tizbac2@gmail.com>
To: pve-devel@lists.proxmox.com
Subject: S3 Support
Date: Tue, 16 Jul 2024 00:20:31 +0200
Message-ID: <CAMLJ59xq0qBDCfVSDyirB8J++SZUSHYJvHdKYr4bOZks8123_w@mail.gmail.com>

Hi, to everyone, i'm interested in native S3 support on proxmox, is anyone
working on it ( to avoid double work )?
Second question, my idea is to reuse existing pbs code but upload chunks
and index to s3, sounds good? The thing puzzling me is that a lot of code
will end up duplicated , when actually only the part uploading to S3
instead of pbs would be different, chunking, fleeching, restoring, backup,
qemu interface would be almost identical

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] S3 Support

[Fabian Grünbichler]

> Tiziano Bacocco via pve-devel <pve-devel@lists.proxmox.com> hat am 16.07.2024 00:20 CEST geschrieben:
> Hi, to everyone, i'm interested in native S3 support on proxmox, is anyone
> working on it ( to avoid double work )?

None of us is, but there is an external developer that is trying to implement something like this. The whole topic might be a better fit for pbs-devel though ;)

> Second question, my idea is to reuse existing pbs code but upload chunks
> and index to s3, sounds good? The thing puzzling me is that a lot of code
> will end up duplicated , when actually only the part uploading to S3
> instead of pbs would be different, chunking, fleeching, restoring, backup,
> qemu interface would be almost identical

https://bugzilla.proxmox.com/show_bug.cgi?id=2943

backing up directly to S3 will be hard for a multitude of reasons:
- you don't want to write individual chunks as objects (too expensive)
- you don't want to slow down the guest because of the speed of the object storage
- ..

instead, a similar approach to the tape backup feature would be needed:
- define retention and bucket allocation policies
- combine metadata (snapshots/..) and chunks of a local datastore in a meaningful way into buckets
- have some sort of index/catalog that tells you which buckets you need to restore which snapshots into a datastore
- ..

hope this makes sense!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel