From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 961561FF16F for <inbox@lore.proxmox.com>; Tue, 27 May 2025 18:09:15 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id CD5B41B80B; Tue, 27 May 2025 18:09:27 +0200 (CEST) Date: Tue, 27 May 2025 12:08:37 -0400 To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= <f.gruenbichler@proxmox.com>, Proxmox VE development discussion <pve-devel@lists.proxmox.com> References: <mailman.34.1748269920.395.pve-devel@lists.proxmox.com> <1594409888.20674.1748329375230@webmail.proxmox.com> In-Reply-To: <1594409888.20674.1748329375230@webmail.proxmox.com> MIME-Version: 1.0 Message-ID: <mailman.67.1748362166.395.pve-devel@lists.proxmox.com> List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Post: <mailto:pve-devel@lists.proxmox.com> From: Andrei Perapiolkin via pve-devel <pve-devel@lists.proxmox.com> Precedence: list Cc: Andrei Perapiolkin <andrei.perepiolkin@open-e.com> X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> Subject: Re: [pve-devel] Volume live migration concurrency Content-Type: multipart/mixed; boundary="===============4140780525329853502==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com> --===============4140780525329853502== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: <andrei.perepiolkin@open-e.com> X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 54340C955F for <pve-devel@lists.proxmox.com>; Tue, 27 May 2025 18:09:26 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2CDEF1B7E6 for <pve-devel@lists.proxmox.com>; Tue, 27 May 2025 18:08:56 +0200 (CEST) Received: from mout.kundenserver.de (mout.kundenserver.de [217.72.192.75]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-devel@lists.proxmox.com>; Tue, 27 May 2025 18:08:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=open-e.com; s=s1-ionos; t=1748362120; x=1748966920; i=andrei.perepiolkin@open-e.com; bh=KJFX+X70Yu5d1W5T8TFL2oKvme+R7m0TRRtt3jTfuUU=; h=X-UI-Sender-Class:Message-ID:Date:MIME-Version:Subject:To: References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding:cc:content-transfer-encoding: content-type:date:from:message-id:mime-version:reply-to:subject: to; b=InzgrXaqiBtoD1JG/CBcIXzQy+DlUcowzWd8utD1OIx+xjxUr7kzYa5K1JG0kA/V hi6vJWYDBKnLqFuBS6xmAQEuXttHsWjX6rXoF4m8vTbn6AnSQ/f6gBNrceEGz9Qru 69Qnj+ISzIcZ3Dvac/G7U49/ewaxCixa8j4jElcH5cPG042Bb4QBBpCvkAr5CPDev poG+yFok7vzspy63i92O056cmBx+n8Tp/CZLX7Ty7Fkb7T0pJjcgof+rBKXGg6a6I Chslwzs8NBriHZD0lHaEOKcSuuworIQZKkXbpEYxZHaWMmnkE6o/c3gjWrQfDgEds TZi54mkMJvf8oyRtTg== X-UI-Sender-Class: 55c96926-9e95-11ee-ae09-1f7a4046a0f6 Received: from [10.137.0.75] ([149.102.246.43]) by mrelayeu.kundenserver.de (mreue109 [212.227.15.183]) with ESMTPSA (Nemesis) id 1MsI0K-1vDIKb3LIo-00yGFa; Tue, 27 May 2025 18:08:40 +0200 Message-ID: <4773cf91-8a0f-453d-b9a1-11dcad1a193f@open-e.com> Date: Tue, 27 May 2025 12:08:37 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [pve-devel] Volume live migration concurrency To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= <f.gruenbichler@proxmox.com>, Proxmox VE development discussion <pve-devel@lists.proxmox.com> References: <mailman.34.1748269920.395.pve-devel@lists.proxmox.com> <1594409888.20674.1748329375230@webmail.proxmox.com> Content-Language: en-US From: Andrei Perapiolkin <andrei.perepiolkin@open-e.com> In-Reply-To: <1594409888.20674.1748329375230@webmail.proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:r/YwjPhnP7hlfJZ3BdEaMBGy/vqJ9di13IOlXMFGj0gWhhyoJ2Q q9q1fIPplxTy0NIAflZ5qX2+cfvILtEm9xVVq5QbWAghL4NZJusZcltHfu0grfC80vjduK/ LtjFOS4hEa/qadP8UvPTVlP08Mn3x4NxuxsL3uvWS4bNg8arRY2zj7goKXx3ejnnHp3RA3+ XBhe3247p+70mlSGkGsqg== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:UuJMVBzxbTs=;4hRj7VDATyx4dSSsr8ridPiFltD b4R0RH0WbbEpBQh3OjYL5cCXyUjujSdyA1CzWyQaUZcgwzXbODyH5Tot8PZrdyzsNIC3YRd6T pKwlyQwyt2+gN6+r9Tw2+MbXBcdHS4smlTuv9/6B/Lltus9eqpPzBlIvz5XSqu09ulEXHtaFi vjHHFYSXb5+0C4uHWLUDCHmW9fjepgfj7MO9t7Z/pTroZSpAFIjxZrfI7chxcb31eOshou03a zGmvnIOcASnIZnE1G01pyYHN/IjFa4xPO6SDa41a4WGVh4aA2eh4J+0mctkUm5crU2rOUzfkh Li/m/bUaV+3SANnH8/+EeE+alz+TXsvX4+XWB4Af2CCOriQvmRuYnM148VKbYygi7za+BQW/A K0vUR0en2AgUGoyUKCCWZ//XpHjcmyC4YdEJoYEGBnFFqWJh2hL5JV0AO6TwLNcRxfRlfGRqo 40dGTm/8Ab0BC4t6iDCEUJTCID7OPgp5zdvP5rmpzuoiNHWCR/PVBOIOniRVljpSNQH2o0Hd1 7d+b2CSF8wTs426GBvLPxNa8l7JSiQGuHk8+R86pZcRIKRBMr72u0O5guz7iLAbGfTlpyHZnw 3JTNwqoZi3xpD1P10jiomFr4+1zjcU/OfQQ56DqLvLGAthsMF8ms7Dfd8/UoGY2968ojgxkZ5 ugpxFOC7DJg1YhwMFj3waw3Rymop101VVVzzbI/DR/h8WIn3hHaGlkVyI6REa4LDFI9ty3cq2 ukEaFbSRpOwU9yvoMdJK21IaqtRsChP0GQO5tfzi5MiFfro7EfxaRC8RHeK/BsUAN80wwEnW1 afNBoyv+4BvCsZw6IEI7hclgMqIoDXmAtYzZcdxLOLAF2KrGyMZXhJGAx4UFlL/6OuBh79ClQ 70YOeIBoFU8T+GK1PyN2kO7F7mUr222EGo2oDhRk4R6gXXB8VKRJzZQUNvQMmYj/wHHj4hRR5 3u0t5oif4xV5R/odRNqlOPPhgMkq3NzsVVUNLVjzSRkhD61TqppgMsJ5uWwqGF/Ae9GWHBtv7 kghRhKkY0wRJyk6kf1YmhynJIR7cJe0S3z989KvFu/j0iZ5MCrudv3KDtOXXStwChUsmtYDlP Nvan2DuBYFHUDepV+np0j7fBdL6Sjj+U8/l4EK2flF7y4CrmyB6933K6IZL5oC6oeYdSQWebw lOVvsEPYu2/JUW14NhXDf5h8y2v41Wmuh7TfbgENEyYKfeqQXkLu+cIa1kMcVop39sZY0dV8q bldb5RpF3XFZR6g6COe0hkSMGRmGQ1WDvS+nf5qScqHRi+I22LbVS/8gWrdCEQlniPTGBCt6O Hqs8hdiqlh7TbQMaiPjPEZ45ZpoeaH7RsOtfF+EQ6D2RSTNDGsvBXn5nbPOyqhWBBwgzpXp0X XxP9fm4nIeZCVlK9J+jc8iraTy03JYvOllo74uuFERYpLG+YEUePrS0aoNWAIWzNFneKEMhHJ +7LGfmQ== X-SPAM-LEVEL: Spam detection results: 0 AWL -0.269 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_HOSTKARMA_BL 1.5 Sender listed in HOSTKARMA-BLACK RCVD_IN_MSPIKE_H5 0.001 Excellent reputation (+5) RCVD_IN_MSPIKE_WL 0.001 Mailspike good senders RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [open-e.com] > 3. In the context of live migration: Will Proxmox skip calling > /deactivate_volume/ for snapshots that have already been activated? > Should the storage plugin explicitly deactivate all snapshots of a > volume during migration? > a live migration is not concerned with snapshots of shared volumes, and = local > volumes are removed on the source node after the migration has finished.= . > > but maybe you could expand this part? My original idea was that since both 'activate_volume' and=20 'deactivate_volume' methods have a 'snapname' argument they would both=20 be used to activate and deactivate snapshots respectivly. And for each snapshot activation, there would be a corresponding=20 deactivation. However, from observing the behavior during migration, I found that=20 'deactivate_volume' is not called for snapshots that were previously=20 activated with 'activate_volume'. Therefore, I assumed that 'deactivate_volume' is responsible for=20 deactivating all snapshots related to the volume that was previously=20 activated. The purpose if this question was to confirm this. From your response I conclude the following: 1. Migration does not manages(i.e. it does not activate or deactivate=20 them=C2=A0 volume snapshots. 2. All volumes are expected to be present across all nodes in cluster,=20 for 'path' function to work. 3. For migration to work volume should be simultaneously present on both= =20 nodes. However, I couldn't find explicit instructions or guides on when and by=20 whom volume snapshot deactivation should be triggered. Is it possible for a volume snapshot to remain active active after=20 volume itself was deactivated? During testing proxmox 8.2 Ive encountered situations when cloning a=20 volume from a snapshot did not resulted in snapshot deactivation. This leads to the creation of 'dangling' snapshots if=C2=A0 the volume is= =20 later migrated. My current understanding is that all assets related to snapshots should=20 to be removed when volume is deactivation, is it correct? Or all volumes and snapshots expected to be present across the entire=20 cluster until they are explicitly deleted? Second option requires additional recommendation on artifact management. May be it should be sent it as an separate email, but draft it here. If all volumes and snapshots are consistently present across entire=20 cluster and their creation/operation results in creation of additional=20 artifacts(such as iSCSI targets, multipath sessions, etc..), then this=20 artifacts should be removed on deletion of associated volume or snapshot. Currently, it is unclear how all nodes in the cluster are notified of=20 such deletion as only one node in the cluster receives 'free_image' or=20 'volume_snapshot_delete'=C2=A0 request. What is a proper way to instruct plugin on other nodes in the cluster=20 that given volume/snapshot is requested for deletion and all artifacts=20 related to it have to be removed? How should the cleanup tasks be triggered across the remaining nodes? I assume that additional service/daemon would be needed to handle such=20 such tasks. In that case, could it leverage the Proxmox Cluster File System=20 (pmxcfs), special the '/etc/pve/priv' directory to coordinate or store=20 state information related to cleanup operations? Andrei On 5/27/25 03:02, Fabian Gr=C3=BCnbichler wrote: >> Andrei Perapiolkin via pve-devel <pve-devel@lists.proxmox.com> hat am 2= 6.05.2025 16:31 CEST geschrieben: >> Hi Proxmox Community, >> >> I'm curious whether there are any standard or guidelines that govern th= e >> order in which the methods: /activate_volume, deactivate_volume, path/ >> are called during VM live migration. >> >> Assuming the storage plugin supports `live migration`: >> >> 1. Can/path/ be called before /activate_volume?/ > yes > >> 2. When /vm /migrates from/node1/ to/node2, /might /activate_volume/ >> on/node2/ be invoked before /deactivate_volume/ has completed on /node1= ? >> / > it has to be, for a live migration both the source VM and the target VM = need > access to the volume. the migration ensures that only either copy/node i= s > writing to a shared volume at any given time. for a local volume, the vo= lumes > are independent anyway. > >> 3. In the context of live migration: Will Proxmox skip calling >> /deactivate_volume/ for snapshots that have already been activated? >> Should the storage plugin explicitly deactivate all snapshots of a >> volume during migration? > a live migration is not concerned with snapshots of shared volumes, and = local > volumes are removed on the source node after the migration has finished.= . > > but maybe you could expand this part? > --===============4140780525329853502== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============4140780525329853502==--