[pve-devel] [Veeam] Veeam change requests?

[pve-devel] [Veeam] Veeam change requests?

[Andreas Neufert via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 12684 bytes --]

From: Andreas Neufert <Andreas.Neufert@veeam.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: [Veeam] Veeam change requests?
Date: Tue, 17 Sep 2024 07:20:21 +0000
Message-ID: <PH0PR14MB42648F4046208E0BE4C61163F4612@PH0PR14MB4264.namprd14.prod.outlook.com>

Hi Proxmox Dev team,

Tim Marx mentioned that you have some insights and change wishes for the Veeam backup processing and that we should reach out to this list. We would be happy to get this feedback here to be able to address it in our code or join a call if this helps.


Mit freundlichen Grüßen and with best regards
Andreas

Andreas Neufert
Veeam Software Group GmbH
Vice President of Product Management, Alliances


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [Veeam] Veeam change requests?

[Dominik Csapak]

On 9/17/24 09:20, Andreas Neufert via pve-devel wrote:
> 
> Hi Proxmox Dev team,
> 

Hi,

> Tim Marx mentioned that you have some insights and change wishes for the Veeam backup processing and that we should reach out to this list. We would be happy to get this feedback here to be able to address it in our code or join a call if this helps.

Thanks for reaching out!

During (very basic & short) testing, i discovered a few things that are problematic from our point 
of view:

* During backup, there is often a longer running connection open to our QMP socket of running VMs
   (/var/run/qemu-server/XXXX.qmp, where XXXX is the vmid). This blocks our management stack from
   doing certain tasks, like start/stop (probably does not matter during backup) but also
   things like the VNC console, etc.

   a better way would be to close the connections as soon as possible instead of keeping them
   open. (Alternatively using our API/CLI could also be done, but I don't know what
   exact QMP commands you're running)

   if you absolutely need a longer running socket, please open a bug report on
   https://bugzilla.proxmox.com so we can discuss and track that there, how we could make
   a socket available that is not used by our stack

* Another thing that I noticed was that it's not really visible if a backup is running
   for a particular VM, so users might accidentally them down (or pause, etc.). Especially
   i think it's bad if the VM is placed under a HA policy that has 'stopped' as target, as
   that will try to stop the VM by itself. (Though this might be a configuration error in itself?)

   A quick way to fix this would be to have a (custom) lock in our VMs. For longer running tasks
   that block a guest, we have a line 'lock: XXXX' in the config that prevents our stack
   from doing most operations.

   Putting that in would be a very short call to our perl code that locks the config locally
   ( `PVE::QemuConfig->lock_config($vmid, $updatefn) ), checks for existing locks,
   updates the config with a new (custom) lock and writes it again.

   Though i must admit, I'm not sure if custom locks outside of our defined ones would work,
   but I'm sure we could add a 'custom' lock that you could use, should my mentioned
   approach not work properly.

* Also, I noticed that when a guest is started from your stack, you modify the QEMU command line a
   bit, namely removing some options that would be necessary to start the VM during the backup.
   Is there a specific reason why you do it this way, instead of starting the VM through
   our API/CLI?


A more general question last: What is the process for our/your users and us if they/we
find a bug? Where can they be reported to you?

I hope this helps

Best regards
Dominik


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [Veeam] Veeam change requests?

[Andreas Neufert via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 17185 bytes --]

From: Andreas Neufert <Andreas.Neufert@veeam.com>
To: Dominik Csapak <d.csapak@proxmox.com>, Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Pavel Tide <Pavel.TIde@veeam.com>
Subject: Re: [pve-devel] [Veeam] Veeam change requests?
Date: Thu, 19 Sep 2024 08:40:19 +0000
Message-ID: <PH0PR14MB42646286C0090066DDDB9EFCF4632@PH0PR14MB4264.namprd14.prod.outlook.com>

Hi Dominik, thanks for your input and feedback.

I CC @Pavel Tide<mailto:Pavel.TIde@veeam.com>, who can answer the code questions.

For the support. Veeam customers can open a support case at https://support.veeam.com<https://support.veeam.com/>

I would like to ask the PVE group to do the same when you identify a bug and select the evaluation product choice from the dropdown.
Then please send me or Pavel the support ticket number so that we can route the ticket differently (out of evaluation support).
The advantage of using the Veeam support portal is that you can upload logs, and we can share fixes with you for testing or similar things.
To start the support ticket with the evaluation product selection, you need to create a free account on veeam.com

If nothing works, Pavel and I can help directly (send mail). We also monitor this list for Veeam issues and questions.

Best regards… Andreas

From: Dominik Csapak <d.csapak@proxmox.com>
Date: Thursday, 19. September 2024 at 09:59
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Cc: Andreas Neufert <Andreas.Neufert@veeam.com>
Subject: Re: [pve-devel] [Veeam] Veeam change requests?
This is the first time you've received an email from this sender d.csapak @ proxmox.com, please exercise caution when clicking on links or opening attachments.

On 9/17/24 09:20, Andreas Neufert via pve-devel wrote:
>
> Hi Proxmox Dev team,
>

Hi,

> Tim Marx mentioned that you have some insights and change wishes for the Veeam backup processing and that we should reach out to this list. We would be happy to get this feedback here to be able to address it in our code or join a call if this helps.

Thanks for reaching out!

During (very basic & short) testing, i discovered a few things that are problematic from our point
of view:

* During backup, there is often a longer running connection open to our QMP socket of running VMs
(/var/run/qemu-server/XXXX.qmp, where XXXX is the vmid). This blocks our management stack from
doing certain tasks, like start/stop (probably does not matter during backup) but also
things like the VNC console, etc.

a better way would be to close the connections as soon as possible instead of keeping them
open. (Alternatively using our API/CLI could also be done, but I don't know what
exact QMP commands you're running)

if you absolutely need a longer running socket, please open a bug report on
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.proxmox.com%2F&data=05%7C02%7CAndreas.Neufert%40veeam.com%7C32edd9e9d4034ac3e8f308dcd880fbd1%7Cba07baab431b49edadd7cbc3542f5140%7C1%7C0%7C638623295714596487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=SF75C1yogf8n9EPMYzwv%2BSFRDx2Y0%2BSruqkgF%2FuDcQU%3D&reserved=0 so we can discuss and track that there, how we could make
a socket available that is not used by our stack

* Another thing that I noticed was that it's not really visible if a backup is running
for a particular VM, so users might accidentally them down (or pause, etc.). Especially
i think it's bad if the VM is placed under a HA policy that has 'stopped' as target, as
that will try to stop the VM by itself. (Though this might be a configuration error in itself?)

A quick way to fix this would be to have a (custom) lock in our VMs. For longer running tasks
that block a guest, we have a line 'lock: XXXX' in the config that prevents our stack
from doing most operations.

Putting that in would be a very short call to our perl code that locks the config locally
( `PVE::QemuConfig->lock_config($vmid, $updatefn) ), checks for existing locks,
updates the config with a new (custom) lock and writes it again.

Though i must admit, I'm not sure if custom locks outside of our defined ones would work,
but I'm sure we could add a 'custom' lock that you could use, should my mentioned
approach not work properly.

* Also, I noticed that when a guest is started from your stack, you modify the QEMU command line a
bit, namely removing some options that would be necessary to start the VM during the backup.
Is there a specific reason why you do it this way, instead of starting the VM through
our API/CLI?


A more general question last: What is the process for our/your users and us if they/we
find a bug? Where can they be reported to you?

I hope this helps

Best regards
Dominik

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [Veeam] Veeam change requests?

[Pavel Tide via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 17723 bytes --]

From: Pavel Tide <Pavel.TIde@veeam.com>
To: Andreas Neufert <Andreas.Neufert@veeam.com>, Dominik Csapak <d.csapak@proxmox.com>, Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: RE: [pve-devel] [Veeam] Veeam change requests?
Date: Mon, 23 Sep 2024 11:33:10 +0000
Message-ID: <IA0PR14MB67672F6D0372302A1F738254936F2@IA0PR14MB6767.namprd14.prod.outlook.com>

Hi Dominik,

For now the best course of action would be to post a message on our forums (forums.veeam.com) in this subsection:

https://forums.veeam.com/kvm-rhv-olvm-proxmox-f62/

I am in the process of arranging some external bug-tracker (we don't have one right now). If you have any preferences please let me know.

As for the question about how we work with QEMU - let me find an answer and I will get back to you shortly.

Thanks!


From: Andreas Neufert <Andreas.Neufert@veeam.com>
Sent: Thursday, September 19, 2024 10:40
To: Dominik Csapak; Proxmox VE development discussion; Pavel Tide
Subject: Re: [pve-devel] [Veeam] Veeam change requests?

Hi Dominik, thanks for your input and feedback.

I CC @Pavel Tide<mailto:Pavel.TIde@veeam.com>, who can answer the code questions.
For the support. Veeam customers can open a support case at https://support.veeam.com<https://support.veeam.com/>

I would like to ask the PVE group to do the same when you identify a bug and select the evaluation product choice from the dropdown.
Then please send me or Pavel the support ticket number so that we can route the ticket differently (out of evaluation support).
The advantage of using the Veeam support portal is that you can upload logs, and we can share fixes with you for testing or similar things.
To start the support ticket with the evaluation product selection, you need to create a free account on veeam.com

If nothing works, Pavel and I can help directly (send mail). We also monitor this list for Veeam issues and questions.

Best regards... Andreas

From: Dominik Csapak <d.csapak@proxmox.com<mailto:d.csapak@proxmox.com>>
Date: Thursday, 19. September 2024 at 09:59
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com<mailto:pve-devel@lists.proxmox.com>>
Cc: Andreas Neufert <Andreas.Neufert@veeam.com<mailto:Andreas.Neufert@veeam.com>>
Subject: Re: [pve-devel] [Veeam] Veeam change requests?
This is the first time you've received an email from this sender d.csapak @ proxmox.com, please exercise caution when clicking on links or opening attachments.

On 9/17/24 09:20, Andreas Neufert via pve-devel wrote:
>
> Hi Proxmox Dev team,
>

Hi,

> Tim Marx mentioned that you have some insights and change wishes for the Veeam backup processing and that we should reach out to this list. We would be happy to get this feedback here to be able to address it in our code or join a call if this helps.

Thanks for reaching out!

During (very basic & short) testing, i discovered a few things that are problematic from our point
of view:

* During backup, there is often a longer running connection open to our QMP socket of running VMs
(/var/run/qemu-server/XXXX.qmp, where XXXX is the vmid). This blocks our management stack from
doing certain tasks, like start/stop (probably does not matter during backup) but also
things like the VNC console, etc.

a better way would be to close the connections as soon as possible instead of keeping them
open. (Alternatively using our API/CLI could also be done, but I don't know what
exact QMP commands you're running)

if you absolutely need a longer running socket, please open a bug report on
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.proxmox.com%2F&data=05%7C02%7CAndreas.Neufert%40veeam.com%7C32edd9e9d4034ac3e8f308dcd880fbd1%7Cba07baab431b49edadd7cbc3542f5140%7C1%7C0%7C638623295714596487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=SF75C1yogf8n9EPMYzwv%2BSFRDx2Y0%2BSruqkgF%2FuDcQU%3D&reserved=0<https://bugzilla.proxmox.com/> so we can discuss and track that there, how we could make
a socket available that is not used by our stack

* Another thing that I noticed was that it's not really visible if a backup is running
for a particular VM, so users might accidentally them down (or pause, etc.). Especially
i think it's bad if the VM is placed under a HA policy that has 'stopped' as target, as
that will try to stop the VM by itself. (Though this might be a configuration error in itself?)

A quick way to fix this would be to have a (custom) lock in our VMs. For longer running tasks
that block a guest, we have a line 'lock: XXXX' in the config that prevents our stack
from doing most operations.

Putting that in would be a very short call to our perl code that locks the config locally
( `PVE::QemuConfig->lock_config($vmid, $updatefn) ), checks for existing locks,
updates the config with a new (custom) lock and writes it again.

Though i must admit, I'm not sure if custom locks outside of our defined ones would work,
but I'm sure we could add a 'custom' lock that you could use, should my mentioned
approach not work properly.

* Also, I noticed that when a guest is started from your stack, you modify the QEMU command line a
bit, namely removing some options that would be necessary to start the VM during the backup.
Is there a specific reason why you do it this way, instead of starting the VM through
our API/CLI?


A more general question last: What is the process for our/your users and us if they/we
find a bug? Where can they be reported to you?

I hope this helps

Best regards
Dominik

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel