Re: [pve-devel] Volume live migration concurrency

[Andrei Perapiolkin via pve-devel <pve-devel@lists.proxmox.com>]

[-- Attachment #1: Type: message/rfc822, Size: 11284 bytes --]

From: Andrei Perapiolkin <andrei.perepiolkin@open-e.com>
To: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] Volume live migration concurrency
Date: Tue, 27 May 2025 12:08:37 -0400
Message-ID: <4773cf91-8a0f-453d-b9a1-11dcad1a193f@open-e.com>

> 3. In the context of live migration: Will Proxmox skip calling
> /deactivate_volume/ for snapshots that have already been activated?
> Should the storage plugin explicitly deactivate all snapshots of a
> volume during migration?

> a live migration is not concerned with snapshots of shared volumes, and local
> volumes are removed on the source node after the migration has finished..
>
> but maybe you could expand this part?

My original idea was that since both 'activate_volume' and 
'deactivate_volume' methods have a 'snapname' argument they would both 
be used to activate and deactivate snapshots respectivly.
And for each snapshot activation, there would be a corresponding 
deactivation.
However, from observing the behavior during migration, I found that 
'deactivate_volume' is not called for snapshots that were previously 
activated with 'activate_volume'.
Therefore, I assumed that 'deactivate_volume' is responsible for 
deactivating all snapshots related to the volume that was previously 
activated.
The purpose if this question was to confirm this.

 From your response I conclude the following:
1. Migration does not manages(i.e. it does not activate or deactivate 
them  volume snapshots.
2. All volumes are expected to be present across all nodes in cluster, 
for 'path' function to work.
3. For migration to work volume should be simultaneously present on both 
nodes.

However, I couldn't find explicit instructions or guides on when and by 
whom volume snapshot deactivation should be triggered.

Is it possible for a volume snapshot to remain active active after 
volume itself was deactivated?

During testing proxmox 8.2 Ive encountered situations when cloning a 
volume from a snapshot did not resulted in snapshot deactivation.
This leads to the creation of 'dangling' snapshots if  the volume is 
later migrated.

My current understanding is that all assets related to snapshots should 
to be removed when volume is deactivation, is it correct?
Or all volumes and snapshots expected to be present across the entire 
cluster until they are explicitly deleted?

Second option requires additional recommendation on artifact management.
May be it should be sent it as an separate email, but draft it here.

If all volumes and snapshots are consistently present across entire 
cluster and their creation/operation results in creation of additional 
artifacts(such as iSCSI targets, multipath sessions, etc..), then this 
artifacts should be removed on deletion of associated volume or snapshot.
Currently, it is unclear how all nodes in the cluster are notified of 
such deletion as only one node in the cluster receives 'free_image' or 
'volume_snapshot_delete'  request.
What is a proper way to instruct plugin on other nodes in the cluster 
that given volume/snapshot is requested for deletion and all artifacts 
related to it have to be removed?

How should the cleanup tasks be triggered across the remaining nodes?

I assume that additional service/daemon would be needed to handle such 
such tasks.
In that case, could it leverage the Proxmox Cluster File System 
(pmxcfs), special the '/etc/pve/priv' directory to coordinate or store 
state information related to cleanup operations?


Andrei

On 5/27/25 03:02, Fabian Grünbichler wrote:
>> Andrei Perapiolkin via pve-devel <pve-devel@lists.proxmox.com> hat am 26.05.2025 16:31 CEST geschrieben:
>> Hi Proxmox Community,
>>
>> I'm curious whether there are any standard or guidelines that govern the
>> order in which the methods: /activate_volume, deactivate_volume, path/
>> are called during VM live migration.
>>
>> Assuming the storage plugin supports `live migration`:
>>
>> 1. Can/path/ be called before /activate_volume?/
> yes
>
>> 2. When /vm /migrates from/node1/ to/node2, /might /activate_volume/
>> on/node2/ be invoked before /deactivate_volume/ has completed on /node1?
>> /
> it has to be, for a live migration both the source VM and the target VM need
> access to the volume. the migration ensures that only either copy/node is
> writing to a shared volume at any given time. for a local volume, the volumes
> are independent anyway.
>
>> 3. In the context of live migration: Will Proxmox skip calling
>> /deactivate_volume/ for snapshots that have already been activated?
>> Should the storage plugin explicitly deactivate all snapshots of a
>> volume during migration?
> a live migration is not concerned with snapshots of shared volumes, and local
> volumes are removed on the source node after the migration has finished..
>
> but maybe you could expand this part?
>


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel