[pve-devel] About PVE Backup Integration Guide

[pve-devel] About PVE Backup Integration Guide

[Prashant Patil via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 11241 bytes --]

From: Prashant Patil <Prashant.Gamepatil@veritas.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>, Jason Voneberstein <Jason.vonEberstein@veritas.com>
Subject: About PVE Backup Integration Guide
Date: Tue, 4 Mar 2025 13:21:17 +0000
Message-ID: <PH0PR20MB4520A4201D4560B18A0C830798C82@PH0PR20MB4520.namprd20.prod.outlook.com>

Hello All,
Hope this email finds you well.

At Arctera/Veritas, we are exploring the addition of support for data protection of Proxmox environments within our backup product. We would like to know if there is a detailed backup/restore API guide available that could serve as a good starting point for our engineering team. Could you please assist us with this? We have already reviewed the REST APIs.

Regards,
Prashant


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] About PVE Backup Integration Guide

[Thomas Lamprecht]

Hello Prashant,

Am 04.03.25 um 16:47 schrieb Prashant Patil:
> Hello All,
> Hope this email finds you well.
> 
> At Arctera/Veritas, we are exploring the addition of support for data protection of Proxmox environments within our backup product. We would like to know if there is a detailed backup/restore API guide available that could serve as a good starting point for our engineering team. Could you please assist us with this? We have already reviewed the REST APIs.

Thanks for your interest in supporting Proxmox VE.

We are in progress of adding a way where external backup provider can
provide a plugin to hook safely and correctly into our backup APIs, see:

https://lore.proxmox.com/pve-devel/20241114150754.374376-1-f.ebner@proxmox.com/

This includes two example plugins to showcase how it would be used.

We would be open to fast-track this effort for inclusion with feedback
from backup vendors. While there are other ways to hook one in, these
are not nice and can cause problems – so in the long term we expect that
most/all backup vendors will switch to this plugin method, at least for
supported setups.

best regards
Thomas


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] About PVE Backup Integration Guide

[Prashant Patil via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 14076 bytes --]

From: Prashant Patil <Prashant.Gamepatil@veritas.com>
To: Thomas Lamprecht <t.lamprecht@proxmox.com>, "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>, Jason Voneberstein <Jason.vonEberstein@veritas.com>, Sudhir Subbarao <Sudhir.Subbarao@veritas.com>
Subject: RE: About PVE Backup Integration Guide
Date: Wed, 5 Mar 2025 06:36:29 +0000
Message-ID: <PH0PR20MB4520446B9B012DE3352A8C1F98CB2@PH0PR20MB4520.namprd20.prod.outlook.com>

Thank you, Thomas, for your prompt response to this.



We would like to know following things:

  1.  How much part of the plugins method is done?
  2.  How soon we can start the integration?
  3.  Is there any plugin method's documentation already available? If yes, could you please share?



Regards,

Prashant



-----Original Message-----
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
Sent: 04 March 2025 10:07 PM
To: Prashant Patil <Prashant.Gamepatil@veritas.com>; pve-devel@lists.proxmox.com
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>; Jason Voneberstein <Jason.vonEberstein@veritas.com>
Subject: Re: About PVE Backup Integration Guide





CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you believe this is a phishing email, use the Report to Cybersecurity icon in Outlook.







Hello Prashant,



Am 04.03.25 um 16:47 schrieb Prashant Patil:

> Hello All,

> Hope this email finds you well.

>

> At Arctera/Veritas, we are exploring the addition of support for data protection of Proxmox environments within our backup product. We would like to know if there is a detailed backup/restore API guide available that could serve as a good starting point for our engineering team. Could you please assist us with this? We have already reviewed the REST APIs.



Thanks for your interest in supporting Proxmox VE.



We are in progress of adding a way where external backup provider can provide a plugin to hook safely and correctly into our backup APIs, see:



https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.proxmox.com%2Fpve-devel%2F20241114150754.374376-1-f.ebner%40proxmox.com%2F&data=05%7C02%7CPrashant.Gamepatil%40veritas.com%7C1816d6bad0c4419d245a08dd5b3adcd0%7Cfc8e13c0422c4c55b3eaca318e6cac32%7C0%7C0%7C638767030578307107%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=X3vGHTDdU8eqfj77XLGcn1myfqvNhPJPaum5dAJcw2Q%3D&reserved=0<https://lore.proxmox.com/pve-devel/20241114150754.374376-1-f.ebner@proxmox.com/>



This includes two example plugins to showcase how it would be used.



We would be open to fast-track this effort for inclusion with feedback from backup vendors. While there are other ways to hook one in, these are not nice and can cause problems - so in the long term we expect that most/all backup vendors will switch to this plugin method, at least for supported setups.



best regards

Thomas



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] About PVE Backup Integration Guide

[Fiona Ebner]

Am 05.03.25 um 07:36 schrieb Prashant Patil via pve-devel:
>   1.  How much part of the plugins method is done?

It is an initial, functional version of the API.

>   2.  How soon we can start the integration?

The patches were not applied yet, feedback is welcome. As Thomas already
said "We would be open to fast-track this effort for inclusion with
feedback from backup vendors". I'll see if I can get around to post a
rebased version this week.

>   3.  Is there any plugin method's documentation already available? If yes, could you please share?

It's part of the patches, see:
https://lore.proxmox.com/pve-devel/20241114150754.374376-10-f.ebner@proxmox.com/

>   1.  How to get allocated blocks/sectors from the snapshot of the raw format virtual disk attached to a running VM (disk present on the block based storage such as lvm, lvm-thin, etc)?
>   2.  How to get allocated blocks/sectors from the snapshot of the vmdk format virtual disk attached to a running VM?
>   3.  As per below link, certain storage types do not support snapshots. In such cases, what is the recommended way to take backup of the running VM?

The block tracking is ideally done via QEMU, then you don't require any
special features for the underlying storages. You can also do dirty
tracking for incremental backups. See the following for how this is done
internally with the proposed API:
https://lore.proxmox.com/pve-devel/20241114150754.374376-4-f.ebner@proxmox.com/
https://lore.proxmox.com/pve-devel/20241114150754.374376-5-f.ebner@proxmox.com/

Best Regards,
Fiona


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] About PVE Backup Integration Guide

[Prashant Patil via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 14809 bytes --]

From: Prashant Patil <Prashant.Gamepatil@veritas.com>
To: Fiona Ebner <f.ebner@proxmox.com>, Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Thomas Lamprecht <t.lamprecht@proxmox.com>
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>, Sudhir Subbarao <Sudhir.Subbarao@veritas.com>, Jason Voneberstein <Jason.vonEberstein@veritas.com>
Subject: RE: [pve-devel] About PVE Backup Integration Guide
Date: Mon, 17 Mar 2025 07:02:47 +0000
Message-ID: <PH0PR20MB4520C688E38C97D5DE5FC25B98DF2@PH0PR20MB4520.namprd20.prod.outlook.com>

> The block tracking is ideally done via QEMU, then you don't require any special features for the underlying storages.

Yes, we were able to get sector map of disks present on 'Directory' storage type. However, for other storage types such as lvm, lvm-thin, zfs which supports raw disk format, here we could get entire disk as allocated which is not the case in real. I could not find much information on this, hence wanted to know whether this by-design behaviour or we are missing something? Pasting output of qemu-img map below for the 2 GB disk on zfs. Have also tried getting map over ndb but got the same result. Is there anything that we are missing here?

root@be-proxmox1:/dev/pve# qemu-img map -f raw --output=json /dev/zvol/zfs1/vm-105-disk-2
[{ "start": 0, "length": 2147483648, "depth": 0, "present": true, "zero": false, "data": true, "compressed": false, "offset": 0}]


> certain storage types do not support snapshots. In such cases, what is the recommended way to take backup of the running VM?

As mentioned earlier, we have found few storage devices which does not support snapshot, but have found that we can take individual disk snapshot through 'blockdev-snapshot-sync'. If we have to take backup of the VM, then are we supposed to use this command to snapshot all VM disks?

Thanks
Prashant

-----Original Message-----
From: Fiona Ebner <f.ebner@proxmox.com>
Sent: 10 March 2025 02:44 PM
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>; Thomas Lamprecht <t.lamprecht@proxmox.com>
Cc: Prashant Patil <Prashant.Gamepatil@veritas.com>; Anuradha Joshi <Anuradha.Joshi@veritas.com>; Sudhir Subbarao <Sudhir.Subbarao@veritas.com>; Jason Voneberstein <Jason.vonEberstein@veritas.com>
Subject: Re: [pve-devel] About PVE Backup Integration Guide


CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you believe this is a phishing email, use the Report to Cybersecurity icon in Outlook.



Am 05.03.25 um 07:36 schrieb Prashant Patil via pve-devel:
>   1.  How much part of the plugins method is done?

It is an initial, functional version of the API.

>   2.  How soon we can start the integration?

The patches were not applied yet, feedback is welcome. As Thomas already said "We would be open to fast-track this effort for inclusion with feedback from backup vendors". I'll see if I can get around to post a rebased version this week.

>   3.  Is there any plugin method's documentation already available? If yes, could you please share?

It's part of the patches, see:
https://lore.proxmox.com/pve-devel/20241114150754.374376-10-f.ebner@proxmox.com/

>   1.  How to get allocated blocks/sectors from the snapshot of the raw format virtual disk attached to a running VM (disk present on the block based storage such as lvm, lvm-thin, etc)?
>   2.  How to get allocated blocks/sectors from the snapshot of the vmdk format virtual disk attached to a running VM?
>   3.  As per below link, certain storage types do not support snapshots. In such cases, what is the recommended way to take backup of the running VM?

The block tracking is ideally done via QEMU, then you don't require any special features for the underlying storages. You can also do dirty tracking for incremental backups. See the following for how this is done internally with the proposed API:
https://lore.proxmox.com/pve-devel/20241114150754.374376-4-f.ebner@proxmox.com/
https://lore.proxmox.com/pve-devel/20241114150754.374376-5-f.ebner@proxmox.com/

Best Regards,
Fiona


This message was sent by an employee of Arctera.


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] About PVE Backup Integration Guide

[Fiona Ebner]

Am 17.03.25 um 08:02 schrieb Prashant Patil:
>> The block tracking is ideally done via QEMU, then you don't require any special features for the underlying storages.
> 
> Yes, we were able to get sector map of disks present on 'Directory' storage type. However, for other storage types such as lvm, lvm-thin, zfs which supports raw disk format, here we could get entire disk as allocated which is not the case in real. I could not find much information on this, hence wanted to know whether this by-design behaviour or we are missing something? Pasting output of qemu-img map below for the 2 GB disk on zfs. Have also tried getting map over ndb but got the same result. Is there anything that we are missing here?
> 
> root@be-proxmox1:/dev/pve# qemu-img map -f raw --output=json /dev/zvol/zfs1/vm-105-disk-2
> [{ "start": 0, "length": 2147483648, "depth": 0, "present": true, "zero": false, "data": true, "compressed": false, "offset": 0}]

That sounds like you want to roll your own external solution instead of
using the backup provider API that is currently being developed. With
that API, you get the images to be backed-up as as NBD exports. For
incremental backup, you can read the dirty bitmap. This can also be done
via NBD. Again, then you don't need to worry about the underlying
storage layer at all to support certain features.

>> certain storage types do not support snapshots. In such cases, what is the recommended way to take backup of the running VM?
> 
> As mentioned earlier, we have found few storage devices which does not support snapshot, but have found that we can take individual disk snapshot through 'blockdev-snapshot-sync'. If we have to take backup of the VM, then are we supposed to use this command to snapshot all VM disks?

Backups in QEMU do not use explicit snapshots. A copy-before-write
filter is inserted on top of the source disk in QEMU's block graph. When
new guest writes happen during backup, old data is first copied away to
a fleecing image (or for regular backup directly to the backup target).
The backup provider API then also inserts a snapshot-access node that is
exported via NBD and allows reading the data from the time of the backup
in a consistent fashion (hence "snapshot-access", it's a
virtual/implicit snapshot).

See also this diagram [0].

> [guest]                   [NBD export]
>    |                            |
>    | root                       | root
>    v                 file       v
> [copy-before-write]<------[snapshot-access]
>    |           |
>    | file      | target
>    v           v
> [active-disk] [temp.qcow2]

Best Regards,
Fiona

[0]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg876056.html


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] About PVE Backup Integration Guide

[Prashant Patil via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 15642 bytes --]

From: Prashant Patil <Prashant.Gamepatil@veritas.com>
To: Fiona Ebner <f.ebner@proxmox.com>, Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Thomas Lamprecht <t.lamprecht@proxmox.com>
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>, Sudhir Subbarao <Sudhir.Subbarao@veritas.com>, Jason Voneberstein <Jason.vonEberstein@veritas.com>
Subject: RE: [pve-devel] About PVE Backup Integration Guide
Date: Mon, 17 Mar 2025 13:30:58 +0000
Message-ID: <PH0PR20MB4520276B3A7061528854180598DF2@PH0PR20MB4520.namprd20.prod.outlook.com>

> That sounds like you want to roll your own external solution instead of using the backup provider API that is currently being developed.
We are happy to integrate with backup provider APIs if they are stable and ready for the integration. On this same thread, I had asked few questions about APIs timelines. Could you please help us to get that info?

> Backups in QEMU do not use explicit snapshots. A copy-before-write filter is inserted on top of the source disk in QEMU's block graph.
What is the API to create copy-before-write snapshot of the disk image? And does it support all formats and storage devices?

Thanks
Prashant

-----Original Message-----
From: Fiona Ebner <f.ebner@proxmox.com>
Sent: 17 March 2025 04:23 PM
To: Prashant Patil <Prashant.Gamepatil@veritas.com>; Proxmox VE development discussion <pve-devel@lists.proxmox.com>; Thomas Lamprecht <t.lamprecht@proxmox.com>
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>; Sudhir Subbarao <Sudhir.Subbarao@veritas.com>; Jason Voneberstein <Jason.vonEberstein@veritas.com>
Subject: Re: [pve-devel] About PVE Backup Integration Guide


CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you believe this is a phishing email, use the Report to Cybersecurity icon in Outlook.



Am 17.03.25 um 08:02 schrieb Prashant Patil:
>> The block tracking is ideally done via QEMU, then you don't require any special features for the underlying storages.
>
> Yes, we were able to get sector map of disks present on 'Directory' storage type. However, for other storage types such as lvm, lvm-thin, zfs which supports raw disk format, here we could get entire disk as allocated which is not the case in real. I could not find much information on this, hence wanted to know whether this by-design behaviour or we are missing something? Pasting output of qemu-img map below for the 2 GB disk on zfs. Have also tried getting map over ndb but got the same result. Is there anything that we are missing here?
>
> root@be-proxmox1:/dev/pve# qemu-img map -f raw --output=json
> /dev/zvol/zfs1/vm-105-disk-2 [{ "start": 0, "length": 2147483648,
> "depth": 0, "present": true, "zero": false, "data": true,
> "compressed": false, "offset": 0}]

That sounds like you want to roll your own external solution instead of using the backup provider API that is currently being developed. With that API, you get the images to be backed-up as as NBD exports. For incremental backup, you can read the dirty bitmap. This can also be done via NBD. Again, then you don't need to worry about the underlying storage layer at all to support certain features.

>> certain storage types do not support snapshots. In such cases, what is the recommended way to take backup of the running VM?
>
> As mentioned earlier, we have found few storage devices which does not support snapshot, but have found that we can take individual disk snapshot through 'blockdev-snapshot-sync'. If we have to take backup of the VM, then are we supposed to use this command to snapshot all VM disks?

Backups in QEMU do not use explicit snapshots. A copy-before-write filter is inserted on top of the source disk in QEMU's block graph. When new guest writes happen during backup, old data is first copied away to a fleecing image (or for regular backup directly to the backup target).
The backup provider API then also inserts a snapshot-access node that is exported via NBD and allows reading the data from the time of the backup in a consistent fashion (hence "snapshot-access", it's a virtual/implicit snapshot).

See also this diagram [0].

> [guest]                   [NBD export]
>    |                            |
>    | root                       | root
>    v                 file       v
> [copy-before-write]<------[snapshot-access]
>    |           |
>    | file      | target
>    v           v
> [active-disk] [temp.qcow2]

Best Regards,
Fiona

[0]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg876056.html


This message was sent by an employee of Arctera.


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] About PVE Backup Integration Guide

[Prashant Patil via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 15401 bytes --]

From: Prashant Patil <Prashant.Gamepatil@veritas.com>
To: Thomas Lamprecht <t.lamprecht@proxmox.com>, "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>, Jason Voneberstein <Jason.vonEberstein@veritas.com>, Sudhir Subbarao <Sudhir.Subbarao@veritas.com>
Subject: RE: About PVE Backup Integration Guide
Date: Wed, 5 Mar 2025 15:44:26 +0000
Message-ID: <PH0PR20MB452048F028DE3978CD3BB9BA98CB2@PH0PR20MB4520.namprd20.prod.outlook.com>

Hi Thomas,
We have few more questions in addition to those mentioned in the previous email.

  1.  How to get allocated blocks/sectors from the snapshot of the raw format virtual disk attached to a running VM (disk present on the block based storage such as lvm, lvm-thin, etc)?
  2.  How to get allocated blocks/sectors from the snapshot of the vmdk format virtual disk attached to a running VM?
  3.  As per below link, certain storage types do not support snapshots. In such cases, what is the recommended way to take backup of the running VM?

https://pve.proxmox.com/wiki/Storage

Could you kindly assist us in obtaining answers to these questions?

Thanks
Prashant

From: Prashant Patil <Prashant.Gamepatil@veritas.com>
Sent: 05 March 2025 12:06 PM
To: Thomas Lamprecht <t.lamprecht@proxmox.com>; pve-devel@lists.proxmox.com
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com>; Jason Voneberstein <Jason.vonEberstein@veritas.com>; Sudhir Subbarao <Sudhir.Subbarao@veritas.com>
Subject: RE: About PVE Backup Integration Guide


Thank you, Thomas, for your prompt response to this.



We would like to know following things:

  1.  How much part of the plugins method is done?
  2.  How soon we can start the integration?
  3.  Is there any plugin method's documentation already available? If yes, could you please share?



Regards,

Prashant



-----Original Message-----
From: Thomas Lamprecht <t.lamprecht@proxmox.com<mailto:t.lamprecht@proxmox.com>>
Sent: 04 March 2025 10:07 PM
To: Prashant Patil <Prashant.Gamepatil@veritas.com<mailto:Prashant.Gamepatil@veritas.com>>; pve-devel@lists.proxmox.com<mailto:pve-devel@lists.proxmox.com>
Cc: Anuradha Joshi <Anuradha.Joshi@veritas.com<mailto:Anuradha.Joshi@veritas.com>>; Jason Voneberstein <Jason.vonEberstein@veritas.com<mailto:Jason.vonEberstein@veritas.com>>
Subject: Re: About PVE Backup Integration Guide





CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you believe this is a phishing email, use the Report to Cybersecurity icon in Outlook.







Hello Prashant,



Am 04.03.25 um 16:47 schrieb Prashant Patil:

> Hello All,

> Hope this email finds you well.

>

> At Arctera/Veritas, we are exploring the addition of support for data protection of Proxmox environments within our backup product. We would like to know if there is a detailed backup/restore API guide available that could serve as a good starting point for our engineering team. Could you please assist us with this? We have already reviewed the REST APIs.



Thanks for your interest in supporting Proxmox VE.



We are in progress of adding a way where external backup provider can provide a plugin to hook safely and correctly into our backup APIs, see:



https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.proxmox.com%2Fpve-devel%2F20241114150754.374376-1-f.ebner%40proxmox.com%2F&data=05%7C02%7CPrashant.Gamepatil%40veritas.com%7C1816d6bad0c4419d245a08dd5b3adcd0%7Cfc8e13c0422c4c55b3eaca318e6cac32%7C0%7C0%7C638767030578307107%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=X3vGHTDdU8eqfj77XLGcn1myfqvNhPJPaum5dAJcw2Q%3D&reserved=0<https://lore.proxmox.com/pve-devel/20241114150754.374376-1-f.ebner@proxmox.com/>



This includes two example plugins to showcase how it would be used.



We would be open to fast-track this effort for inclusion with feedback from backup vendors. While there are other ways to hook one in, these are not nice and can cause problems - so in the long term we expect that most/all backup vendors will switch to this plugin method, at least for supported setups.



best regards

Thomas



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel