From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <uwe.sauter.de@gmail.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DA83DFF2D for <pve-user@lists.proxmox.com>; Tue, 25 Jul 2023 13:48:25 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BA94619F29 for <pve-user@lists.proxmox.com>; Tue, 25 Jul 2023 13:48:25 +0200 (CEST) Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-user@lists.proxmox.com>; Tue, 25 Jul 2023 13:48:25 +0200 (CEST) Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-52227142a27so3303504a12.1 for <pve-user@lists.proxmox.com>; Tue, 25 Jul 2023 04:48:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690285698; x=1690890498; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:reply-to:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=x0VVTSBSRzt/yJ3ZyxKbmBCg1cp6xzXjTrN2yCafQ7M=; b=Ka/A90QNy7KbICGOOgRVkgq8T1LMTI/NMwSBXVCfstC33yYqgnABhd2uH/POBPSe8s iQPSHccNkIRc4r076SP0qvNdxVMGVmIKu9Kyxrbg0UP9ITvUqaD17rr4bG5cr/LBZhNs K9K0q30xsvno/ENOAZYhH0z8Br3rnL4oMGygH6nYCykhWLK1nzXEK/5TneTO7bHGd28v e217ywLsldyWISbbRDPYTAt9BnxpEzOM9/jIWafRWpAGTnqBOsOpcRbA3S4mtvbHndEM x7teX1RnNe1Q+NdL7vx+ecimPiYj7Vvtbi/ahID1X3spxNSlt3cnZvdH9MWzNxsqNeoI d77Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690285698; x=1690890498; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:reply-to:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=x0VVTSBSRzt/yJ3ZyxKbmBCg1cp6xzXjTrN2yCafQ7M=; b=GeONmleJYfOF0oRRVUVKGxGKyNcPJ4uqb2D1VKu/Ah5ZjIUc5nMzJ4wRieJsxEctto h1FoFWXnu8HvUyD0TwpL2FOmTCEqgAh7EBumBPxJSeqcDiORl+RyAzJfNy3664OX5PXF ZLOuALcFhV0WlV+QHfkQ809k7XUXQoa1c1+i+O75GQojHSJfLw1wX3xvCpT1dhIytt9V rHSBp4/NfIKCiLb86s6qYqvIDNp4/VVt5P+cY7M/pe+1NrHwkK2oGJ6Z/n7IvL/XEMr8 rGSOQ3yJTfTRK7aeCVJJzWqKTh+5SM8pW/4mnn1JDJ7Aop7CNZLGJYYMSgg7CmmfX2rU 6s6Q== X-Gm-Message-State: ABy/qLaleUFx7RMmhIot/4BihRcpYbBzgVSmRR1trrIobLQ8f+YwfKJo KL6k6PBr3QS2nves+SOU0yw= X-Google-Smtp-Source: APBJJlGwSHj5+xJNxDYRGSKuZrkgdphtGZFZzW1alVyXLXSka2NFjDg1PNaDRxBtE2yBXiquCD4bSQ== X-Received: by 2002:aa7:d7d0:0:b0:522:3ebc:84b9 with SMTP id e16-20020aa7d7d0000000b005223ebc84b9mr2756073eds.24.1690285698302; Tue, 25 Jul 2023 04:48:18 -0700 (PDT) Received: from ?IPV6:2a02:8070:a280:2d80:5605:dbff:fe76:161d? ([2a02:8070:a280:2d80:5605:dbff:fe76:161d]) by smtp.googlemail.com with ESMTPSA id q2-20020a056402032200b0051d9de03516sm7483807edw.52.2023.07.25.04.48.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 25 Jul 2023 04:48:17 -0700 (PDT) Message-ID: <337ffcc2-2893-93ad-d7b7-8c665b565a89@gmail.com> Date: Tue, 25 Jul 2023 13:48:16 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: uwe.sauter.de@gmail.com Content-Language: de-DE, en-US To: Alwin Antreich <alwin@antreich.com>, Proxmox VE user list <pve-user@lists.proxmox.com> References: <d94e484f-7504-fb07-80a1-13fd52518594@gmail.com> <2b5b83bb-c90e-b6dd-4b15-a57414b42542@gmail.com> <dc743429b8e92c12ec74c8844605f4b1@antreich.com> From: Uwe Sauter <uwe.sauter.de@gmail.com> In-Reply-To: <dc743429b8e92c12ec74c8844605f4b1@antreich.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.140 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [ceph.com, proxmox.com, diskmanage.pm] Subject: Re: [PVE-User] DeviceMapper devices get filtered by Proxmox X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Tue, 25 Jul 2023 11:48:25 -0000 Hi Alwin, Am 25.07.23 um 12:40 schrieb Alwin Antreich: > Hi Uwe, > > July 25, 2023 9:24 AM, "Uwe Sauter" <uwe.sauter.de@gmail.com> wrote: > >> So, I've been looking further into this and indeed, there seem to be very strict filters regarding >> the block device names that Proxmox allows to be used. >> >> /usr/share/perl5/PVE/Diskmanage.pm >> >> 512 # whitelisting following devices >> 513 # - hdX ide block device >> 514 # - sdX scsi/sata block device >> 515 # - vdX virtIO block device >> 516 # - xvdX: xen virtual block device >> 517 # - nvmeXnY: nvme devices >> 518 # - cciss!cXnY cciss devices >> 519 print Dumper($dev); >> 520 return if $dev !~ m/^(h|s|x?v)d[a-z]+$/ && >> 521 $dev !~ m/^nvme\d+n\d+$/ && >> 522 $dev !~ m/^cciss\!c\d+d\d+$/; >> >> I don't understand all the consequences of allowing ALL ^dm-\d+$ devices but with proper filtering >> it should be possible to allow multipath devices (and given that there might be udev rules that >> create additinal symlinks below /dev, each device's name should be resolved to its canonical name >> before checking). > It is also a matter of ceph support [0]. Aside the extra complexity, using the amount of HDDs is not a good use-case for virtualization. And HDDs definitely need the DB/WAL on a separate device (60x disks -> 5x NVMe). Well, if the documentation is to be trusted, there is multipath support since Octupus. My use-case is not hyper-converged virtualization; I simply am using Proxmox due to its good UI and integration of Ceph (and because it does not rely on containers to deploy Ceph). I am aware that HDDs will need some amount of flash but I do have a couple of SAS-SSDs at hand that I can put into the JBODs. And currently all this is just a proof of concept. > Best to set it up with ceph-volume directly. See the forum post [1] for the experience of other users. Thanks for the link though I have to support the arguments of the forum members that multipath is an enterprise feature that should be supported by an enterprise-class virtualization solution. Best, Uwe > Cheers, > Alwin > > [0] https://docs.ceph.com/en/latest/ceph-volume/lvm/prepare/#multipath-support > [1] https://forum.proxmox.com/threads/ceph-with-multipath.70813/