public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>,
	Stoiko Ivanov <s.ivanov@proxmox.com>
Subject: [pve-devel] applied: [PATCH storage v2 2/3] zfspoolplugin: check if mounted instead of imported
Date: Fri, 19 Feb 2021 15:24:05 +0100	[thread overview]
Message-ID: <9d6fcfe8-fc16-d6e4-e4fb-61c2c5c0d466@proxmox.com> (raw)
In-Reply-To: <20210219124544.12593-3-s.ivanov@proxmox.com>

On 19.02.21 13:45, Stoiko Ivanov wrote:
> This patch addresses an issue we recently saw on a production machine:
> * after booting a ZFS pool failed to get imported (due to an empty
>   /etc/zfs/zpool.cache)
> * pvestatd/guest-startall eventually tried to import the pool
> * the pool was imported, yet the datasets of the pool remained
>   not mounted
> 
> A bit of debugging showed that `zpool import <poolname>` is not
> atomic, in fact it does fork+exec `mount` with appropriate parameters.
> If an import ran longer than the hardcoded timeout of 15s, it could
> happen that the pool got imported, but the zpool command (and its
> forks) got terminated due to timing out.
> 
> reproducing this is straight-forward by setting (drastic) bw+iops
> limits on a guests' disk (which contains a zpool) - e.g.:
> `qm set 100 -scsi1 wd:vm-100-disk-1,iops_rd=10,iops_rd_max=20,\
> iops_wr=15,iops_wr_max=20,mbps_rd=10,mbps_rd_max=15,mbps_wr=10,\
> mbps_wr_max=15`
> afterwards running `timeout 15 zpool import <poolname>` resulted in
> that situation in the guest on my machine
> 
> The patch changes the check in activate_storage for the ZFSPoolPlugin,
> to check if any dataset below the 'pool' (which can also be a sub-dataset)
> is mounted by parsing /proc/mounts:
> * this is cheaper than running `zfs get` or `zpool list`
> * it catches a properly imported and mounted pool in case the
>   root-dataset has 'canmount' set to off (or noauto), as long
>   as any dataset below is mounted
> 
> After trying to import the pool, we also run `zfs mount -a` (in case
> another check of /proc/mounts fails).
> 
> Potential for regression:
> * running `zfs mount -a` is problematic, if a dataset is manually
>   umounted after booting (without setting 'canmount')
> * a pool without any mounted dataset (no mountpoint property set and
>   only zvols) - will result in repeated calls to `zfs mount -a`
> 
> both of the above seem unlikely and should not occur, if using our
> tooling.
> 
> Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
> ---
>  PVE/Storage/ZFSPoolPlugin.pm | 27 ++++++++++++++++++++++++---
>  1 file changed, 24 insertions(+), 3 deletions(-)
> 
>

applied, thanks!




  reply	other threads:[~2021-02-19 14:24 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-19 12:45 [pve-devel] [PATCH storage v2 0/3] " Stoiko Ivanov
2021-02-19 12:45 ` [pve-devel] [PATCH storage v2 1/3] zfspoolplugin: activate_storage: minor cleanup Stoiko Ivanov
2021-02-19 13:18   ` [pve-devel] applied: " Thomas Lamprecht
2021-02-19 12:45 ` [pve-devel] [PATCH storage v2 2/3] zfspoolplugin: check if mounted instead of imported Stoiko Ivanov
2021-02-19 14:24   ` Thomas Lamprecht [this message]
2021-02-19 12:45 ` [pve-devel] [PATCH storage v2 3/3] zfspoolplugin: check if imported before importing Stoiko Ivanov
2021-02-19 15:28   ` [pve-devel] applied: " Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9d6fcfe8-fc16-d6e4-e4fb-61c2c5c0d466@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    --cc=s.ivanov@proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal