From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <Bryan@bryanfields.net> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id ADB659DDFC for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 11:26:28 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 85B107675 for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 11:25:58 +0100 (CET) Received: from morty.keekles.org (Morty.keekles.org [199.47.174.151]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 11:25:54 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by morty.keekles.org (Postfix) with ESMTP id AFDFF19E1BCF for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 10:16:12 +0000 (UTC) Received: from morty.keekles.org ([127.0.0.1]) by localhost (morty.keekles.org [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 2o-_0-vj2KbO for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 10:16:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by morty.keekles.org (Postfix) with ESMTP id 3D5B719E1C11 for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 10:16:08 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 morty.keekles.org 3D5B719E1C11 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bryanfields.net; s=909DCF92-EFE7-11EB-9235-648EB8AF1B81; t=1698574568; bh=XX7hoQVr/TXxUAyymK8EqXcs0TgUBK5WrmM+w1AJZgo=; h=Message-ID:Date:MIME-Version:To:From; b=IY2WUVBKzN1XrOt64XudRVHlelgS0YM0VtMwyZr+ohWKK262a56k9eZEnWg6peZAz 3bAEdJEq0uQpzeyGW62uwJxNVrvvrJvN0aUrDCD2greF2lq2cZ7YFYkUTtm2aND1rK s9fMDFpqZUqNpjHFarA9PML+fyKKjZaKKA18nAKycluphwPlmSbkBsB9tQmS8eC4De EtsdsCUP0oPwE7Gy2AWPNkIb5b/RydX4zYiKjOL/QtvH/GaA9h8m4CNpuzDleUPz2+ v7U3dQcz2UOnrUGJgh4PxVGyM9y3Wk4qP1Oi7Gaw/7ewAKm4wMvd+KMBlC1TWz0Iif xhxX8tu4KSi2Q== X-Virus-Scanned: amavisd-new at morty.keekles.org Received: from morty.keekles.org ([127.0.0.1]) by localhost (morty.keekles.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ipLrxsJYzyws for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 10:16:08 +0000 (UTC) Received: from [192.168.128.105] (static-47-206-239-202.tamp.fl.frontiernet.net [47.206.239.202]) by morty.keekles.org (Postfix) with ESMTPSA id 0A27019E1C0F for <pve-user@lists.proxmox.com>; Sun, 29 Oct 2023 10:16:08 +0000 (UTC) Message-ID: <8bbed148-012b-cd0f-cf44-49f4e891464d@bryanfields.net> Date: Sun, 29 Oct 2023 06:16:07 -0400 MIME-Version: 1.0 User-Agent: Mutt/1.12.0 (2019-05-25) Content-Language: en-US To: pve-user@lists.proxmox.com From: Bryan Fields <Bryan@bryanfields.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.037 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_MISSING 0.1 Missing DMARC policy SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record T_SPF_PERMERROR 0.01 SPF: test of record failed (permerror) Subject: [PVE-User] Shared ZFS storage that supports LXC's X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Sun, 29 Oct 2023 10:26:28 -0000 I've been working to migrate my servers to a shared storage from a local-zfs. My backend is a Linux server with 24 16T SAS drives in ZFS as follows: 4x (6d raidz2) + 3x mirror 4T NVME + 375G Optane split 32G/312G ZIL/L2ARC 768G of ram, and 1/2 that is for zfs arc. All connected via 2x10g lag with jumbo frames to the other servers. I have been working with the ZFS over ISCSI backend for my VM's and it works very well. The only issue is there's not any support for multipath, but with a 10g network, I'm not running into limits here for practical purposes. I do get a warning in proxmox when moving a VM into it or making a snapshot: Warning: volblocksize (8192) is much less than the minimum allocation unit (32768), which wastes at least 75% of space. To reduce wasted space, use a larger volblocksize (32768 is recommended), fewer dRAID data disks per group, or smaller sector size (ashift). I can't find exactly what this is referring to or how to fix it. Does anyone have insight into this message? With the LXC's I've found they don't support this backend storage. (and it's not mentioned in the docs) I assume this is do to them needing a filesystem, not a block device. My option here would be to run NFS for shared storage, but this loses the ability to do snapshots (a must have). LVM would work, but it's not able to be shared. I was thinking it might make sense to do this as a per LXC NFS mount via ZFS, and then the PVE node makes a new dataset in zfs on the shared storage server via ssh for that lxc. This is basically how the zfs over iscsi is handled today, as how I understand it. Has anyone solved this? I'd like an option here to migrate LXC's from local to shared storage on a linux/zfs server. Thanks, -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net