[pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[DERUMIER, Alexandre]

Hi,

Currently, proxmox backup are using copy-before-write to have snapshot 
backup.

This works great until you have a slow  or remote storage with higher 
latencies than the vm storage.

This can slowdown a lot the vm is a lot of writes occur when backup is 
running. (or even can hang the vm if backup storage goes down during the 
backup)


qemu backup support since 1-2 years some kind of local write cache, 
called "fleecing".

When backup is running, if a block is not yet backuped, it's going to a 
temp local disk image.

Some improvements have been made in qemu 7.0, previously only qcow2 was 
allowed for the temp disk, now it's working with any disk

https://github.com/qemu/qemu/commit/af5bcd775f3115e4c1b7715920a67b31b119de30


Could it be possible to implement same king of cache for proxmox backup ?

I think qemu implementation is a simple block filter (It's even possible 
to expose a virtual snasphot), so maybe it could be easy to implement it ?


This is really a blocker for me,I can't use pbs because I'm using nvme 
is production, and a 7200k hdd backup in a remote 200km site with 5ms 
latency.

If I could use temp nvme cache on the hypervisor, it could be wonderfull :)


I had opened a bugzilla some week ago with same kind of local cache idea 
(didn't known about qemu fleecing at this time, seem the way to go)

https://bugzilla.proxmox.com/show_bug.cgi?id=4136

Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[Dietmar Maurer]

> This is really a blocker for me,I can't use pbs because I'm using nvme 
> is production, and a 7200k hdd backup in a remote 200km site with 5ms 
> latency.

Why don't you use a local(fast) PBS instance, then sync to the slow remote?

Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[DERUMIER, Alexandre]

Le 31/07/22 à 18:19, Dietmar Maurer a écrit :
>> This is really a blocker for me,I can't use pbs because I'm using nvme
>> is production, and a 7200k hdd backup in a remote 200km site with 5ms
>> latency.
> Why don't you use a local(fast) PBS instance, then sync to the slow remote?
>
Hi Dietmar.

Can I use a small local fast PBS instance without need to keep the full 
datastore chunks ?

I have 300TB nvme in production, I don't want to buy 300TB nvme for backup.

I known that I can keep more retentions on remote storage slow, but what 
about the local fast pbs ?

Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[Laurent GUERBY]

On Sun, 2022-07-31 at 16:49 +0000, DERUMIER, Alexandre wrote:
> Le 31/07/22 à 18:19, Dietmar Maurer a écrit :
> > > This is really a blocker for me,I can't use pbs because I'm using
> > > nvme
> > > is production, and a 7200k hdd backup in a remote 200km site with
> > > 5ms
> > > latency.
> > 
> > Why don't you use a local(fast) PBS instance, then sync to the slow
> > remote?
> > 
> 
> Hi Dietmar.
> 
> Can I use a small local fast PBS instance without need to keep the
> full 
> datastore chunks ?
> 
> I have 300TB nvme in production, I don't want to buy 300TB nvme for
> backup.
> 
> I known that I can keep more retentions on remote storage slow, but
> what 
> about the local fast pbs ?
> 

Hi Alexandre,

I had the exact same issue:

https://forum.proxmox.com/threads/partial-sync-job-pre-prune.106441/

Here is the bugzilla:

https://bugzilla.proxmox.com/show_bug.cgi?id=3701

So no AFAIK there's no way to do a partial sync. A tool to sync a
specific backup (instead of a group) and a bit of scripting would do
the job.

Sincerely,

Laurent

> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[DERUMIER, Alexandre]

Le 31/07/22 à 19:47, Laurent GUERBY a écrit :

On Sun, 2022-07-31 at 16:49 +0000, DERUMIER, Alexandre wrote:


Le 31/07/22 à 18:19, Dietmar Maurer a écrit :


This is really a blocker for me,I can't use pbs because I'm using
nvme
is production, and a 7200k hdd backup in a remote 200km site with
5ms
latency.



Why don't you use a local(fast) PBS instance, then sync to the slow
remote?




Hi Dietmar.

Can I use a small local fast PBS instance without need to keep the
full
datastore chunks ?

I have 300TB nvme in production, I don't want to buy 300TB nvme for
backup.

I known that I can keep more retentions on remote storage slow, but
what
about the local fast pbs ?




Hi Alexandre,

I had the exact same issue:

https://antiphishing.cetsi.fr/proxy/v3?i=SGI0YVJGNmxZNE90Z2thMFYLWSxJOfIERJocpmb73Vs&r=SW5LV3JodE9QZkRVZ3JEYYKpfBJeBDlAX9EWSgcxKTOskC-MKbYiKWCY7M6ocEtDq8CeXBnAzGsMk-U5K5P6Yg&f=bnJjU3hQT3pQSmNQZVE3aO0r-ZB14cskqerzYUDW5Vk-5C50pA9QKkzP7UNmmVTG&u=https%3A//forum.proxmox.com/threads/partial-sync-job-pre-prune.106441/&k=dFBm

Here is the bugzilla:

https://antiphishing.cetsi.fr/proxy/v3?i=SGI0YVJGNmxZNE90Z2thMFYLWSxJOfIERJocpmb73Vs&r=SW5LV3JodE9QZkRVZ3JEYYKpfBJeBDlAX9EWSgcxKTOskC-MKbYiKWCY7M6ocEtDq8CeXBnAzGsMk-U5K5P6Yg&f=bnJjU3hQT3pQSmNQZVE3aO0r-ZB14cskqerzYUDW5Vk-5C50pA9QKkzP7UNmmVTG&u=https%3A//bugzilla.proxmox.com/show_bug.cgi%3Fid%3D3701&k=dFBm

So no AFAIK there's no way to do a partial sync. A tool to sync a
specific backup (instead of a group) and a bit of scripting would do
the job.

Sincerely,

Laurent

Hi Laurent !

I think that my need is a little bit different.

looking at your forum message

"have it sync to our current existing primary PBS server (SSD). The NVME server will have less space than the SSD server and not enough space to have a copy of all backups of the primary,"


I think It's not the same. I need the inverse.  My primary storage need to be fast and small, and secondary synced remote is big && slow.









_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com<mailto:pve-devel@lists.proxmox.com>
https://antiphishing.cetsi.fr/proxy/v3?i=SGI0YVJGNmxZNE90Z2thMFYLWSxJOfIERJocpmb73Vs&r=SW5LV3JodE9QZkRVZ3JEYYKpfBJeBDlAX9EWSgcxKTOskC-MKbYiKWCY7M6ocEtDq8CeXBnAzGsMk-U5K5P6Yg&f=bnJjU3hQT3pQSmNQZVE3aO0r-ZB14cskqerzYUDW5Vk-5C50pA9QKkzP7UNmmVTG&u=https%3A//lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel&k=dFBm

Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[DERUMIER, Alexandre]

Le 31/07/22 à 18:49, DERUMIER, Alexandre a écrit :
> Le 31/07/22 à 18:19, Dietmar Maurer a écrit :
>>> This is really a blocker for me,I can't use pbs because I'm using nvme
>>> is production, and a 7200k hdd backup in a remote 200km site with 5ms
>>> latency.
>> Why don't you use a local(fast) PBS instance, then sync to the slow remote?
>>
> Hi Dietmar.
>
> Can I use a small local fast PBS instance without need to keep the full
> datastore chunks ?
>
> I have 300TB nvme in production, I don't want to buy 300TB nvme for backup.
>
> I known that I can keep more retentions on remote storage slow, but what
> about the local fast pbs ?
>
>
Maybe also, currently if you pbs server is crashing/shutdown/halt/... 
when a backup is running,

the vm writes are totally frozen.

Or if network problem occur, it can hang the vm too.

That's why I think than a local cache (could be optionnal) could be a 
great improvment.



I found doc about fleecing,

technally, it's just exposing a new blockdev inside qemu, like a virtual 
frozen snapshot.

So I think it could work with proxmox backup code too.

https://www.mail-archive.com/qemu-devel@nongnu.org/msg876056.html


####create the fleecing device

qmp: transaction [
    block-dirty-bitmap-add {node: disk0, name: bitmap0, persistent: true}
    blockdev-add* {node-name: tmp-protocol, driver: file, filename: 
temp.qcow2}
    blockdev-add {node-name: tmp, driver: qcow2, file: tmp-protocol}
    blockdev-add {node-name: cbw, driver: copy-before-write, file: disk0,
target: tmp}
    blockdev-replace** {parent-type: qdev, qdev-id: sda, new-child: cbw}
    blockdev-add {node-name: acc, driver: snapshot-access, file: cbw}
]



#### launch qemu backup (push model) --> should use proxmox backup code 
here instead


# Add target node. Here is qcow2 added, but it may be nbd node or 
something else
     blockdev-add {node-name: target-protocol, driver: file, filename:
target.qcow2}
     blockdev-add {node-name: target, driver: qcow2, file: target-protocol}

# Start backup
     blockdev-backup {device: acc, target: target, ...}
]

Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[Dietmar Maurer]

> Can I use a small local fast PBS instance without need to keep the full 
> datastore chunks ?
> 
> I have 300TB nvme in production, I don't want to buy 300TB nvme for backup.

no, sorry.

Do you want to use Ceph as temp backup storage, or simply an additional (node local) nvme?

Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"

[DERUMIER, Alexandre]

Le 1/08/22 à 10:31, Dietmar Maurer a écrit :
>> Can I use a small local fast PBS instance without need to keep the full
>> datastore chunks ?
>>
>> I have 300TB nvme in production, I don't want to buy 300TB nvme for backup.
> no, sorry.
>
> Do you want to use Ceph as temp backup storage, or simply an additional (node local) nvme?
>
Both methods are ok for me, I just need a fast storage somewhere on the 
primary datacenter.

If I can reuse my ceph cluster, why not, don't need to buy new local nvme.

local nvme could avoid to add load to the ceph cluster, but it's really 
marginal.