From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 729371FF0AB for ; Wed, 07 Oct 2026 13:21:20 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2419F21219; Wed, 07 Oct 2026 13:21:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linbit.com; s=google; t=1791372059; x=1791976859; darn=lists.proxmox.com; h=user-agent:content-disposition:content-type:mime-version:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=NqP3nX+UYGEi4Vyh/08dh2IxrbKrPWbfdpNhA50UsvA=; b=P474ZjBr954RVoFTXS238Nt32NTBV0gixiZumr0vPEIobI2oCovE/xl6nJ/NwskZct 9M48vhnkbMSvrcTOxkvuLh0eh0rjhaDeYJxzmed3eESyu5aE3PPaGdpjqOOkbhf8vdG4 hL5YEs78ogL+302Z/RNyuRGCh+FB/ZnI5xJsVeNuZe6Uq6b3zwsoaujwm+xZ/wPuZL52 cT15wDmPcDukgBUrYkslJ9C6NkEUQ4btvsDzHZmz9PgcRxQyg5kamwd995DmMIPAftTm 4skF/81gIeTDzB/jn8TiCTU7cWeeyjYVa0xN24+akF8xMbQQKY2o9OeBUBBjAVNfZmLP fX0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791372059; x=1791976859; h=user-agent:content-disposition:content-type:mime-version:message-id :subject:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to:content-type; bh=NqP3nX+UYGEi4Vyh/08dh2IxrbKrPWbfdpNhA50UsvA=; b=OEdpJGm5AAggwhNwCwhSLAxxmqZA0bfrZntMbWX0kn0OnzKukcazonXRI82hXByW5+ buu9g+J/ySli/TOsDXYUF4JuuQT5HkKRp9c0l136kKKdB8s9T7YwcinhAHQkR/t4/o0/ YBLe28Inv4dXYn9ocQlsV/3DfUGLnfe3iHO01JjJfa58xpkyoPrVghibxY8HNr48lAKe wVMjLJxU/TVdR4cydtOHrKOhSNDP4ReUqJs2RYKSZXdwv53joO+zrJWdx+TdNZRPTDdK K5i2zwFC4XhmX45RJji8OlHpD0FFetJXg6JvSRMilOhO9O8srhCy2ywucaiwiK6xOP/N /dyg== X-Gm-Message-State: AFq9FYIRI3GqJPezaoV5Lv2DVUPijHLsWMXBcfQt8xS49Y6IvZX5Iqmv kqY2GVBSD2RFAQ7q9eYXvyHPdsZordYYEh0gJ7KBMU6YmsvKT28IAFbmSqRs4uxK2huSCGi0Su4 IwkMfbLlbOA== X-Gm-Gg: AYBFou2WZZpejzOY/Uc+aQ0J/neFSOYuwUyIogUBA8c4r3YCW1CVota7Yb6Tp7ofF+s 9KqCvbekwtMwAYsDFAchq++SEbbwjF5oVLlcAeWMg7QFu8yBEOmed4rx1Xbqd+CD515dylhTdqQ 4L/idCbfywp5oHX6rXaMcCLYi0DEStOPsdC/ALL/FFRT1a04Mig6sAKGIlelub/SHxE/4sZR7u+ Vtq6FfgBXtKJ0Tm4AHPyZpPXjGiTJYQt+FTcOxeJO1whLS+++OHlxG8UTga8c8B028S+qKw9rRe W38VmJjYt5k5SCxOHHrNz/hls/5j+4FjsRComVLbqRZf6JUWqYj9/kVmSnMfCW95wReZtAHKCLg Y/y7HpE945siAlCo9uZoShNkT+FMYXWsTmkxhNHucO0TQQ0hcxfEarXo3q+Jb1KLC9jRjn6VwSD HwRsDMaal1FmSlNDHiPhLvcJhdKjTXEYpVlfsUaQa69hRWtgJo/sierTw2eJuN2Bux5Ptrw4vHb 99GaZ3oVnzsGaV6PfgJZa8kmrC6253ZDcYso0zjIl1KKEmewK3l X-Received: by 2002:a05:6512:31c2:b0:5b2:f426:9c63 with SMTP id 2adb3069b0e04-5bcd06c10f6mr722637e87.22.1791372058744; Wed, 07 Oct 2026 04:20:58 -0700 (PDT) Date: Wed, 7 Oct 2026 13:20:55 +0200 From: Roland Kammerer To: pve-devel@lists.proxmox.com Subject: [RFC storage/qemu-server] add 'live-migration' hint to volume (de)activation Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/2.3.3 (2026-06-12) X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: XER7RF3SSQUP4DCMVATLYBABWHED3NL7 X-Message-ID-Hash: XER7RF3SSQUP4DCMVATLYBABWHED3NL7 X-MailFrom: roland.kammerer@linbit.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, Disclaimer: parts of this RFC have been written with the help of an LLM. This results in a relatively verbose RFC, but I kept the details in the hope they can be useful. At this time I'm mainly interested if the idea for hints when a live migration starts/ends is a good idea in general or not. I marked LLM assisted/generated sections where I think it adds some benefit for the reviewers. With the hints mechanism introduced in storage plugin APIVER 13 [0] there is now a clean, well-defined channel to pass contextual information from the upper layers down to storage plugins on volume activation. I would like to propose a new hint that tells a plugin that a volume (de)activation happens in the context of a live migration, including the role of the local node. Motivation ========== Some storage technologies need to know about the concurrent-access window that a live migration opens. The concrete case is DRBD via the external linstor-proxmox plugin: DRBD normally allows exactly one Primary (writable) node per resource. During a live migration, the target QEMU process opens the disks read-write while the source QEMU still has them open, so the resource temporarily needs to allow two Primaries. Today there is no signal for this, so the plugin has to either keep allow-two-primaries enabled permanently (current situation, undesirable, as it weakens split-brain protection during normal operation) or guess from resource state (racy). Other plugins with single-writer semantics face the same problem. Another problem with setting allow-two-primaries permanently is that it allows admins, scripts,... to open the in-use device on a peer node and unwillingly altering data by accident. Specifically for DRBD we only allow two Primaries when using DRBD protocol C (the one with the strongest guarantees). Sometime people would like to use weaker guarantees (i.e., protocol A and B), which we can only allow when we sacrifice live migration as protocols A and B don't allow two Primaries at all. If we would know the live migration window, we could temporarily upgrade the connections between these 2 nodes to protocol C and (also temporarily) allow-two-primaries. LLM generated, unverified: Current behavior (as of qemu-server master) ====================================================================== - Migration start: the target node activates all volumes in vm_start_nolock() before command line generation. This call site already passes hints (generate_storage_hints()), and the incoming migration context is in scope there. - Migration end (success): the source node deactivates the volumes via phase3_cleanup() -> vm_stop() -> vm_stop_cleanup() -> deactivate_volumes(). No hints support in that path today. - Migration end (abort after target start): the source VM keeps running and never deactivates; instead the *target* tears down its VM via vm_stop() (which already carries $migratedfrom on this path) -> vm_stop_cleanup() -> deactivate_volumes(). So the concurrent-access window is opened by exactly one call (hinted activate on the target) and closed by exactly one call (deactivate on the source on success, or on the target on abort). The proposal is to mark all three. LLM assisted, reviewed to my best knowledge: Proposal ===================================================== 1) pve-storage: add a new hint property 'live-migration' => { type => 'string', enum => ['incoming', 'outgoing'], optional => 1, description => "The volume belongs to a guest that is currently being live-migrated. The value denotes the role of the local node in the migration.", }, Per the APIVER 13 rules, adding a hint itself needs no version bump. 2) pve-storage: add a $hints parameter to the deactivate_volume() plugin method (appended after $cache) and to PVE::Storage::deactivate_volumes(), analogous to activate_volume(). This is a backward-compatible signature extension: APIVER 17 with APIAGE increased, existing plugins can simply ignore the additional parameter. 3) qemu-server: pass the hint at the three sites above, guarded by PVE::Storage::Plugin::is_hint_supported() as usual, plus a versioned dependency bump on libpve-storage-perl: - vm_start_nolock(): 'incoming' on activate_volumes() when starting for an incoming live migration. - source, after successful migration: 'outgoing' on the deactivation. - target, when cleaning up a failed incoming migration: 'incoming' on the deactivation. For the two deactivation sites this means threading the migration context through vm_stop()/vm_stop_cleanup() (which already know $migratedfrom on the abort path), or alternatively explicit deactivate_volumes() calls in the QemuMigrate cleanup phases. I am happy to go either way, opinions welcome. LLM assisted, reviewed to my best knowledge: Semantics for plugins ================================================================== An activation hinted 'incoming' means a concurrent-activation window opens: the volume is (or will shortly be) simultaneously active on the migration source. A deactivation hinted 'incoming' or 'outgoing' means that window is closed. A plugin that needs to relax single-writer enforcement can key it off exactly these calls. In line with the existing hints contract, this stays best-effort: hints are not guaranteed on every call, and a node crash mid-migration means the closing deactivation may never happen. Plugins must therefore remain correct without the hint and self-heal (e.g. linstor-proxmox would record that it relaxed the setting and clear it on the next unhinted (de)activation of the volume). The hint makes the common path robust and removes the need for permanently relaxed settings; it is not a transactional API. Consumer ======== linstor-proxmox would use the hint to temporarily allow two Primaries for the specific resource during the migration window (LINSTOR already knows which node is Primary and can adjust settings per connection), instead of requiring the current permanently-enabled configuration. plugin change alongside as a demonstration. If the general direction is agreeable, I could try to follow up with patches. If you think the description is good enough and the implementation is better done by PVE devs, that certainly is fine as well. Looking forward to your feedback. Best regards, Roland [0] https://lore.proxmox.com/pve-devel/20251031103709.60233-1-f.weber@proxmox.com/