[pve-devel] [PATCH storage/zfsonlinux v2 0/3] fix #4997: lvm: avoid autoactivating (new) LVs after boot

* [pve-devel] [PATCH storage/zfsonlinux v2 0/3] fix #4997: lvm: avoid autoactivating (new) LVs after boot
@ 2025-03-07  9:52 Friedrich Weber
  2025-03-07  9:52 ` [pve-devel] [PATCH zfsonlinux v2 1/3] zfs-initramfs: add patch to avoid activating all LVs at boot Friedrich Weber
                   ` (3 more replies)
  0 siblings, 4 replies; 12+ messages in thread
From: Friedrich Weber @ 2025-03-07  9:52 UTC (permalink / raw)
  To: pve-devel

# Summary

With default settings, LVM autoactivates LVs when it sees a new VG, e.g. after
boot or iSCSI login. In a cluster with guest disks on a shared LVM VG (e.g. on
top of iSCSI/Fibre Channel (FC)/direct-attached SAS), this can indirectly cause
guest creation or migration to fail. See bug #4997 [1] and patch #3 for
details.

The goal of this series is to avoid autoactivating LVs that hold guest disks in
order to fix #4997. With this series applied, (newly created) LVs should only
be active on the node that hosts the corresponding guest.

# Difference to v1

v1 [0] went with an approach that creates new LVs with the "activation skip"
flag set, which means lvchange will only activate the LV if the `-K` flag is
passed. This approach requires a two-step patch apply process to not break
mixed-version clusters: With PVE 8.4, we would add a change that passes the
`-K` flag when activating, and only with PVE 9 we would also create new LVs
with the "activation skip" flag. This approach works but is relatively complex.

# This v2 approach

This v2 goes a different route: Instead of "activation skip" flag, we use the
the `--setautoactivation n` flag for LVs. This flag is available in Debian's
LVM since Bookworm and only concerns autoactivation, not normal activation as
performed by our LVM plugin. Hence, it is enough to create new LVs with
`--setautoactivation n`, we don't require any two-step patch apply process. We
may still need to wait for PVE 9 to apply this though (see next section).
Considering this, the advantage over v1 may appear quite small, though I still
find v2's approach preferable because the `--setautoactivation` flag seems less
invasive than the activation skip flag.

# Mixed 7/8 cluster

Unfortunately, we need to consider the mixed-version cluster between PVE 7 and
PVE 8 because PVE 7/Bullseye's LVM does not know `--setautoactivation`. A user
upgrading from PVE 7 will temporarily have a mixed 7/8 cluster. Once this
series is applied, the PVE 8 nodes will create new LVs with
`--setautoactivation n`, which the PVE 7 nodes do not know. In my tests, the
PVE 7 nodes can read/interact with such LVs just fine, *but*: As soon as a PVE
7 node creates a new (unrelated) LV, the `--setautoactivation n` flag is reset
to default `y` on *all* LVs of the VG. I presume this is because creating a new
LV rewrites metadata, and the PVE 7 LVM doesn't write out the
`--setautoactivation n` flag. I imagine (have not tested) this will cause
problems on a mixed cluster.

Hence, it may be safer to hold off on applying patches #2/#3 until PVE 9,
because then we can be sure all nodes run at least PVE 8.

# Interaction with zfs-initramfs

One complication is that zfs-initramfs ships an initramfs-tools script that
unconditionally activates *all* VGs that are visible at boot time, see [2].
This is the case for local VGs, but also for shared VGs on FC/SAS. Patch #2 of
this series fixes this by making the script perform autoactivation instead,
which honors the `--setautoactivation` flag. The patch is necessary to fix
#4997 on FC/SAS-attached LUNs too.

# Bonus fix for FC/SAS multipath+LVM issue

As it turns out, this series seems to additionally fix an issue on hosts with
LVM on FC/SAS-attached LUNs *with multipath* where LVM would report "Device
mismatch detected" warnings because the LVs are activated too early in the boot
process before multipath is available. Our current suggested workaround is to
install multipath-tools-boot [2]. With this series applied, this shouldn't be
necessary anymore, as (newly created) LVs are not auto-activated after boot.

# LVM-thick/LVM-thin

Note that this change affects all LVs on LVM-thick, not just ones on shared
storage. As a result, also on single-node hosts, local guest disk LVs on
LVM-thick will not be automatically active after boot anymore (after applying
all patches of this series). Guest disk LVs on LVM-thin will still be
auto-activated, but since LVM-thin storage is necessarily local, we don't run
into #4997 here.

# Transition to LVs with `--setautoactivation n`

Both v1 and v2 approaches only take effect for new LVs, so we should probably
also have pve8to9 check for guest disks on (shared?) LVM that have
autoactivation enabled, and suggest to the user to manually disable
autoactivation on the LVs, or even the entire VG if it holds only PVE-managed
LVs.

We could implement something on top to make the transition smoother, some ideas:

- When activating existing LVs, check the auto activation flag, and if auto
  activation is still enabled, disable it.

- When creating a new VG via the LVM storage plugin or the /disks/lvm API
  endpoints, disable autoactivation for the whole VG. But this may become
  confusing as then we'd deal with the flag on both LVs and VGs. Also, especially
  for FC/SAS-attached LUNs, users may not use the GUI/API and instead create the
  VG via the command line. We could adjust our guides to use `vgcreate
  --setautoactivation n` [4], but not all users may follow these guides.

# Patch summary

- Patch #1 patches zfsonlinux-initramfs to avoid unconditional activation
  of all VGs/LVs on boot
- Patch #2 is preparation
- Patch #3 makes the LVM plugin create new LVs with `--setautoactivation n`

[0] v1: https://lore.proxmox.com/pve-devel/20240111150332.733635-1-f.weber@proxmox.com/
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=4997
[2] https://lore.proxmox.com/pve-devel/174a386d-2c1e-4023-acf0-2c3d0ea1417d@proxmox.com/
[3] https://pve.proxmox.com/mediawiki/index.php?title=Multipath&oldid=12039#%22Device_mismatch_detected%22_warnings
[4] https://pve.proxmox.com/mediawiki/index.php?title=Multipath&oldid=12039#LVM_on_top_of_a_LUN

zfsonlinux:

Friedrich Weber (1):
  zfs-initramfs: add patch to avoid activating all LVs at boot

 ...s-use-LVM-autoactivation-for-activat.patch | 49 +++++++++++++++++++
 debian/patches/series                         |  1 +
 2 files changed, 50 insertions(+)
 create mode 100644 debian/patches/0012-contrib-initramfs-use-LVM-autoactivation-for-activat.patch

pve-storage:

Friedrich Weber (2):
  lvm: create: use multiple lines for lvcreate commandline definition
  fix #4997: lvm: create: disable autoactivation for new logical volumes

 src/PVE/Storage/LVMPlugin.pm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

Summary over all repositories:
  3 files changed, 59 insertions(+), 1 deletions(-)

-- 
Generated by git-murpp 0.7.3

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

^ permalink raw reply	[flat|nested] 12+ messages in thread