From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
	by lore.proxmox.com (Postfix) with ESMTPS id 961561FF16F
	for <inbox@lore.proxmox.com>; Tue, 27 May 2025 18:09:15 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id CD5B41B80B;
	Tue, 27 May 2025 18:09:27 +0200 (CEST)
Date: Tue, 27 May 2025 12:08:37 -0400
To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= <f.gruenbichler@proxmox.com>,
 Proxmox VE development discussion <pve-devel@lists.proxmox.com>
References: <mailman.34.1748269920.395.pve-devel@lists.proxmox.com>
 <1594409888.20674.1748329375230@webmail.proxmox.com>
In-Reply-To: <1594409888.20674.1748329375230@webmail.proxmox.com>
MIME-Version: 1.0
Message-ID: <mailman.67.1748362166.395.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Andrei Perapiolkin via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Andrei Perapiolkin <andrei.perepiolkin@open-e.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: Re: [pve-devel] Volume live migration concurrency
Content-Type: multipart/mixed; boundary="===============4140780525329853502=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>

--===============4140780525329853502==
Content-Type: message/rfc822
Content-Disposition: inline

Return-Path: <andrei.perepiolkin@open-e.com>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits))
	(No client certificate requested)
	by lists.proxmox.com (Postfix) with ESMTPS id 54340C955F
	for <pve-devel@lists.proxmox.com>; Tue, 27 May 2025 18:09:26 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id 2CDEF1B7E6
	for <pve-devel@lists.proxmox.com>; Tue, 27 May 2025 18:08:56 +0200 (CEST)
Received: from mout.kundenserver.de (mout.kundenserver.de [217.72.192.75])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange ECDHE (prime256v1) server-signature RSA-PSS (2048 bits) server-digest SHA256)
	(No client certificate requested)
	by firstgate.proxmox.com (Proxmox) with ESMTPS
	for <pve-devel@lists.proxmox.com>; Tue, 27 May 2025 18:08:55 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=open-e.com;
	s=s1-ionos; t=1748362120; x=1748966920;
	i=andrei.perepiolkin@open-e.com;
	bh=KJFX+X70Yu5d1W5T8TFL2oKvme+R7m0TRRtt3jTfuUU=;
	h=X-UI-Sender-Class:Message-ID:Date:MIME-Version:Subject:To:
	 References:From:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:cc:content-transfer-encoding:
	 content-type:date:from:message-id:mime-version:reply-to:subject:
	 to;
	b=InzgrXaqiBtoD1JG/CBcIXzQy+DlUcowzWd8utD1OIx+xjxUr7kzYa5K1JG0kA/V
	 hi6vJWYDBKnLqFuBS6xmAQEuXttHsWjX6rXoF4m8vTbn6AnSQ/f6gBNrceEGz9Qru
	 69Qnj+ISzIcZ3Dvac/G7U49/ewaxCixa8j4jElcH5cPG042Bb4QBBpCvkAr5CPDev
	 poG+yFok7vzspy63i92O056cmBx+n8Tp/CZLX7Ty7Fkb7T0pJjcgof+rBKXGg6a6I
	 Chslwzs8NBriHZD0lHaEOKcSuuworIQZKkXbpEYxZHaWMmnkE6o/c3gjWrQfDgEds
	 TZi54mkMJvf8oyRtTg==
X-UI-Sender-Class: 55c96926-9e95-11ee-ae09-1f7a4046a0f6
Received: from [10.137.0.75] ([149.102.246.43]) by mrelayeu.kundenserver.de
 (mreue109 [212.227.15.183]) with ESMTPSA (Nemesis) id
 1MsI0K-1vDIKb3LIo-00yGFa; Tue, 27 May 2025 18:08:40 +0200
Message-ID: <4773cf91-8a0f-453d-b9a1-11dcad1a193f@open-e.com>
Date: Tue, 27 May 2025 12:08:37 -0400
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [pve-devel] Volume live migration concurrency
To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= <f.gruenbichler@proxmox.com>,
 Proxmox VE development discussion <pve-devel@lists.proxmox.com>
References: <mailman.34.1748269920.395.pve-devel@lists.proxmox.com>
 <1594409888.20674.1748329375230@webmail.proxmox.com>
Content-Language: en-US
From: Andrei Perapiolkin <andrei.perepiolkin@open-e.com>
In-Reply-To: <1594409888.20674.1748329375230@webmail.proxmox.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:r/YwjPhnP7hlfJZ3BdEaMBGy/vqJ9di13IOlXMFGj0gWhhyoJ2Q
 q9q1fIPplxTy0NIAflZ5qX2+cfvILtEm9xVVq5QbWAghL4NZJusZcltHfu0grfC80vjduK/
 LtjFOS4hEa/qadP8UvPTVlP08Mn3x4NxuxsL3uvWS4bNg8arRY2zj7goKXx3ejnnHp3RA3+
 XBhe3247p+70mlSGkGsqg==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:UuJMVBzxbTs=;4hRj7VDATyx4dSSsr8ridPiFltD
 b4R0RH0WbbEpBQh3OjYL5cCXyUjujSdyA1CzWyQaUZcgwzXbODyH5Tot8PZrdyzsNIC3YRd6T
 pKwlyQwyt2+gN6+r9Tw2+MbXBcdHS4smlTuv9/6B/Lltus9eqpPzBlIvz5XSqu09ulEXHtaFi
 vjHHFYSXb5+0C4uHWLUDCHmW9fjepgfj7MO9t7Z/pTroZSpAFIjxZrfI7chxcb31eOshou03a
 zGmvnIOcASnIZnE1G01pyYHN/IjFa4xPO6SDa41a4WGVh4aA2eh4J+0mctkUm5crU2rOUzfkh
 Li/m/bUaV+3SANnH8/+EeE+alz+TXsvX4+XWB4Af2CCOriQvmRuYnM148VKbYygi7za+BQW/A
 K0vUR0en2AgUGoyUKCCWZ//XpHjcmyC4YdEJoYEGBnFFqWJh2hL5JV0AO6TwLNcRxfRlfGRqo
 40dGTm/8Ab0BC4t6iDCEUJTCID7OPgp5zdvP5rmpzuoiNHWCR/PVBOIOniRVljpSNQH2o0Hd1
 7d+b2CSF8wTs426GBvLPxNa8l7JSiQGuHk8+R86pZcRIKRBMr72u0O5guz7iLAbGfTlpyHZnw
 3JTNwqoZi3xpD1P10jiomFr4+1zjcU/OfQQ56DqLvLGAthsMF8ms7Dfd8/UoGY2968ojgxkZ5
 ugpxFOC7DJg1YhwMFj3waw3Rymop101VVVzzbI/DR/h8WIn3hHaGlkVyI6REa4LDFI9ty3cq2
 ukEaFbSRpOwU9yvoMdJK21IaqtRsChP0GQO5tfzi5MiFfro7EfxaRC8RHeK/BsUAN80wwEnW1
 afNBoyv+4BvCsZw6IEI7hclgMqIoDXmAtYzZcdxLOLAF2KrGyMZXhJGAx4UFlL/6OuBh79ClQ
 70YOeIBoFU8T+GK1PyN2kO7F7mUr222EGo2oDhRk4R6gXXB8VKRJzZQUNvQMmYj/wHHj4hRR5
 3u0t5oif4xV5R/odRNqlOPPhgMkq3NzsVVUNLVjzSRkhD61TqppgMsJ5uWwqGF/Ae9GWHBtv7
 kghRhKkY0wRJyk6kf1YmhynJIR7cJe0S3z989KvFu/j0iZ5MCrudv3KDtOXXStwChUsmtYDlP
 Nvan2DuBYFHUDepV+np0j7fBdL6Sjj+U8/l4EK2flF7y4CrmyB6933K6IZL5oC6oeYdSQWebw
 lOVvsEPYu2/JUW14NhXDf5h8y2v41Wmuh7TfbgENEyYKfeqQXkLu+cIa1kMcVop39sZY0dV8q
 bldb5RpF3XFZR6g6COe0hkSMGRmGQ1WDvS+nf5qScqHRi+I22LbVS/8gWrdCEQlniPTGBCt6O
 Hqs8hdiqlh7TbQMaiPjPEZ45ZpoeaH7RsOtfF+EQ6D2RSTNDGsvBXn5nbPOyqhWBBwgzpXp0X
 XxP9fm4nIeZCVlK9J+jc8iraTy03JYvOllo74uuFERYpLG+YEUePrS0aoNWAIWzNFneKEMhHJ
 +7LGfmQ==
X-SPAM-LEVEL: Spam detection results:  0
	AWL                    -0.269 Adjusted score from AWL reputation of From: address
	BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
	DKIM_SIGNED               0.1 Message has a DKIM or DK signature, not necessarily valid
	DKIM_VALID               -0.1 Message has at least one valid DKIM or DK signature
	DKIM_VALID_AU            -0.1 Message has a valid DKIM or DK signature from author's domain
	DKIM_VALID_EF            -0.1 Message has a valid DKIM or DK signature from envelope-from domain
	DMARC_PASS               -0.1 DMARC pass policy
	POISEN_SPAM_PILL          0.1 Meta: its spam
	POISEN_SPAM_PILL_1        0.1 random spam to be learned in bayes
	POISEN_SPAM_PILL_3        0.1 random spam to be learned in bayes
	RCVD_IN_DNSWL_NONE     -0.0001 Sender listed at https://www.dnswl.org/, no trust
	RCVD_IN_HOSTKARMA_BL      1.5 Sender listed in HOSTKARMA-BLACK
	RCVD_IN_MSPIKE_H5       0.001 Excellent reputation (+5)
	RCVD_IN_MSPIKE_WL       0.001 Mailspike good senders
	RCVD_IN_VALIDITY_CERTIFIED_BLOCKED  0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked.  See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information.
	RCVD_IN_VALIDITY_RPBL_BLOCKED  0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked.  See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information.
	RCVD_IN_VALIDITY_SAFE_BLOCKED  0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked.  See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information.
	SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
	SPF_PASS               -0.001 SPF: sender matches SPF record
	URIBL_BLOCKED           0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked.  See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [open-e.com]

> 3. In the context of live migration: Will Proxmox skip calling
> /deactivate_volume/ for snapshots that have already been activated?
> Should the storage plugin explicitly deactivate all snapshots of a
> volume during migration?

> a live migration is not concerned with snapshots of shared volumes, and =
local
> volumes are removed on the source node after the migration has finished.=
.
>
> but maybe you could expand this part?

My original idea was that since both 'activate_volume' and=20
'deactivate_volume' methods have a 'snapname' argument they would both=20
be used to activate and deactivate snapshots respectivly.
And for each snapshot activation, there would be a corresponding=20
deactivation.
However, from observing the behavior during migration, I found that=20
'deactivate_volume' is not called for snapshots that were previously=20
activated with 'activate_volume'.
Therefore, I assumed that 'deactivate_volume' is responsible for=20
deactivating all snapshots related to the volume that was previously=20
activated.
The purpose if this question was to confirm this.

 From your response I conclude the following:
1. Migration does not manages(i.e. it does not activate or deactivate=20
them=C2=A0 volume snapshots.
2. All volumes are expected to be present across all nodes in cluster,=20
for 'path' function to work.
3. For migration to work volume should be simultaneously present on both=
=20
nodes.

However, I couldn't find explicit instructions or guides on when and by=20
whom volume snapshot deactivation should be triggered.

Is it possible for a volume snapshot to remain active active after=20
volume itself was deactivated?

During testing proxmox 8.2 Ive encountered situations when cloning a=20
volume from a snapshot did not resulted in snapshot deactivation.
This leads to the creation of 'dangling' snapshots if=C2=A0 the volume is=
=20
later migrated.

My current understanding is that all assets related to snapshots should=20
to be removed when volume is deactivation, is it correct?
Or all volumes and snapshots expected to be present across the entire=20
cluster until they are explicitly deleted?

Second option requires additional recommendation on artifact management.
May be it should be sent it as an separate email, but draft it here.

If all volumes and snapshots are consistently present across entire=20
cluster and their creation/operation results in creation of additional=20
artifacts(such as iSCSI targets, multipath sessions, etc..), then this=20
artifacts should be removed on deletion of associated volume or snapshot.
Currently, it is unclear how all nodes in the cluster are notified of=20
such deletion as only one node in the cluster receives 'free_image' or=20
'volume_snapshot_delete'=C2=A0 request.
What is a proper way to instruct plugin on other nodes in the cluster=20
that given volume/snapshot is requested for deletion and all artifacts=20
related to it have to be removed?

How should the cleanup tasks be triggered across the remaining nodes?

I assume that additional service/daemon would be needed to handle such=20
such tasks.
In that case, could it leverage the Proxmox Cluster File System=20
(pmxcfs), special the '/etc/pve/priv' directory to coordinate or store=20
state information related to cleanup operations?


Andrei

On 5/27/25 03:02, Fabian Gr=C3=BCnbichler wrote:
>> Andrei Perapiolkin via pve-devel <pve-devel@lists.proxmox.com> hat am 2=
6.05.2025 16:31 CEST geschrieben:
>> Hi Proxmox Community,
>>
>> I'm curious whether there are any standard or guidelines that govern th=
e
>> order in which the methods: /activate_volume, deactivate_volume, path/
>> are called during VM live migration.
>>
>> Assuming the storage plugin supports `live migration`:
>>
>> 1. Can/path/ be called before /activate_volume?/
> yes
>
>> 2. When /vm /migrates from/node1/ to/node2, /might /activate_volume/
>> on/node2/ be invoked before /deactivate_volume/ has completed on /node1=
?
>> /
> it has to be, for a live migration both the source VM and the target VM =
need
> access to the volume. the migration ensures that only either copy/node i=
s
> writing to a shared volume at any given time. for a local volume, the vo=
lumes
> are independent anyway.
>
>> 3. In the context of live migration: Will Proxmox skip calling
>> /deactivate_volume/ for snapshots that have already been activated?
>> Should the storage plugin explicitly deactivate all snapshots of a
>> volume during migration?
> a live migration is not concerned with snapshots of shared volumes, and =
local
> volumes are removed on the source node after the migration has finished.=
.
>
> but maybe you could expand this part?
>


--===============4140780525329853502==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

--===============4140780525329853502==--