From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C3F72FDBD for ; Tue, 25 Jul 2023 09:24:33 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id ACF13166EF for ; Tue, 25 Jul 2023 09:24:33 +0200 (CEST) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 25 Jul 2023 09:24:33 +0200 (CEST) Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-51ff0e3d8c1so7402397a12.0 for ; Tue, 25 Jul 2023 00:24:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690269866; x=1690874666; h=content-transfer-encoding:in-reply-to:references:to:from :content-language:subject:reply-to:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=NoCrBaj3IyC9ZWgWmOnG4mQt05/5oeM/0ulnA6p/9WE=; b=htjMcng0TapTN3PTQ6t4mc/bQdwdPNruuDlIGOUnwjFYyNeA8UJD09kf6qlwR1A5JS FLGPOZv+bD65oEIiLawWgeqkdT54V5e19xDv2O/1qWw0TbcUiXF/b7X0aaC4S3KSiQ67 9EMWNp7MnTg1/PJ0MMEpZ7p2CnriWpbLqkVk+7/T+1wnSuI9I37O7tYEdkg2IP0IQGtq 0dUBbIlvro3NmaXxEod1+2Ils0BmhHn/xblgf4CLs/kxz8k6mkOgAsgbXt+R2WGWd+rp irj964HuuNtWNMq5t+ONvBlkDhRiTVPBVGd55fvTc982/IrHTCYxf0P3G6MWhAnyRxEB 3VKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690269866; x=1690874666; h=content-transfer-encoding:in-reply-to:references:to:from :content-language:subject:reply-to:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NoCrBaj3IyC9ZWgWmOnG4mQt05/5oeM/0ulnA6p/9WE=; b=ZnxxMBt+CKOmqyu14JDqKYRNpboFBnME7XZyjkZZ5KaRTspv826KS3DkZiwd0R0bx/ WfEaRgoZWtyUKg/R3O1VdpDJ38ynEBm+50iz4MY9Ttg9bv4sK7l0w2prEQZlYwWA/Ti8 OxwAK6BcOkQMfb8WZ9HZJfM+rTd01RHWJeoePnTtX85iOSHA2hlb1TD9bFapnv+cBRYp XG4M5eeJFQfQyUG48UkQVyO0+nyU6Tys2pTgvfzpCH2NEjEsJxjizbcYmlzLOWPDc5y/ E0RBKEdMxS3FTv4Ke6szbN6coTAeG6af3foKYHItmREeBUn0XpZpk1L3sOsqYzySYRze yDeA== X-Gm-Message-State: ABy/qLaL/siR9Tbo/uE+sBuI3OfY/9YH7m8tdsYqj+B3u3kIIW2zw3Dk cQfGD3i8TOtP694jyf0CDvYxu+aFKDA= X-Google-Smtp-Source: APBJJlHLJVGuCEsobmwMsiKnA+TvHoiaIZ4hkxLzxAHrIHpdHZA7+pMDCLbf441ZQjo3WnNOBlqi/w== X-Received: by 2002:aa7:d1d5:0:b0:522:55bf:21af with SMTP id g21-20020aa7d1d5000000b0052255bf21afmr413017edp.7.1690269866465; Tue, 25 Jul 2023 00:24:26 -0700 (PDT) Received: from ?IPV6:2a02:8070:a280:2d80:5605:dbff:fe76:161d? ([2a02:8070:a280:2d80:5605:dbff:fe76:161d]) by smtp.googlemail.com with ESMTPSA id v8-20020aa7d808000000b00521d1c34b23sm7137250edq.83.2023.07.25.00.24.25 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 25 Jul 2023 00:24:25 -0700 (PDT) Message-ID: Date: Tue, 25 Jul 2023 09:24:22 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: uwe.sauter.de@gmail.com Content-Language: de-DE From: Uwe Sauter To: PVE User List References: <2b5b83bb-c90e-b6dd-4b15-a57414b42542@gmail.com> In-Reply-To: <2b5b83bb-c90e-b6dd-4b15-a57414b42542@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.160 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [diskmanage.pm] Subject: Re: [PVE-User] DeviceMapper devices get filtered by Proxmox X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jul 2023 07:24:33 -0000 So, I've been looking further into this and indeed, there seem to be very strict filters regarding the block device names that Proxmox allows to be used. /usr/share/perl5/PVE/Diskmanage.pm 512 # whitelisting following devices 513 # - hdX ide block device 514 # - sdX scsi/sata block device 515 # - vdX virtIO block device 516 # - xvdX: xen virtual block device 517 # - nvmeXnY: nvme devices 518 # - cciss!cXnY cciss devices 519 print Dumper($dev); 520 return if $dev !~ m/^(h|s|x?v)d[a-z]+$/ && 521 $dev !~ m/^nvme\d+n\d+$/ && 522 $dev !~ m/^cciss\!c\d+d\d+$/; I don't understand all the consequences of allowing ALL ^dm-\d+$ devices but with proper filtering it should be possible to allow multipath devices (and given that there might be udev rules that create additinal symlinks below /dev, each device's name should be resolved to its canonical name before checking). To give an example I have sdc as an internal disk that contains a Bluestore OSD: ls -l /dev/mapper/ | grep dm-0 lrwxrwxrwx 1 root root 7 Jul 22 23:23 ceph--aa6b72f7--f185--44b4--9922--6ae4e6278d10-osd--block--b0a60266--90c0--444f--ae12--328ecfebd87d -> ../dm-0 Devices sde, sdaw, sdco, sdeg are the same SAS disk and form multipath device dm-2: # ls -la /dev/mapper/ | grep dm-2$ lrwxrwxrwx 1 root root 7 Jul 22 23:23 35000cca26a7402e4 -> ../dm-2 # multipath -ll | grep -A6 'dm-2 ' 35000cca26a7402e4 dm-2 HGST,HUH721010AL5200 size=9.1T features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 1:0:0:0 sde 8:64 active ready running |- 1:0:45:0 sdaw 67:0 active ready running |- 5:0:0:0 sdco 69:192 active ready running `- 5:0:45:0 sdeg 128:128 active ready running Both dm-0 and dm-2 are device mapper devices but it is possible to get their type: # dmsetup status /dev/dm-0 0 3750739968 linear # dmsetup status /dev/dm-2 0 19532873728 multipath 2 0 0 0 1 1 A 0 4 2 8:64 A 0 0 1 67:0 A 0 0 1 69:192 A 0 0 1 128:128 A 0 0 1 Another way, possibly less exact, to filter is if /sys/block/dm-X/slaves/ contains more than one entry. # ls -l /sys/block/dm-0/slaves total 0 lrwxrwxrwx 1 root root 0 Jul 25 09:01 sdc -> ../../../../pci0000:00/0000:00:11.4/ata3/host3/target3:0:0/3:0:0:0/block/sdc # ls -l /sys/block/dm-2/slaves total 0 lrwxrwxrwx 1 root root 0 Jul 25 09:03 sdaw -> ../../../../pci0000:80/0000:80:02.0/0000:82:00.0/host1/port-1:1/expander-1:1/port-1:1:0/end_device-1:1:0/target1:0:45/1:0:45:0/block/sdaw lrwxrwxrwx 1 root root 0 Jul 25 09:03 sdco -> ../../../../pci0000:80/0000:80:03.0/0000:83:00.0/host5/port-5:0/expander-5:0/port-5:0:0/end_device-5:0:0/target5:0:0/5:0:0:0/block/sdco lrwxrwxrwx 1 root root 0 Jul 25 09:03 sde -> ../../../../pci0000:80/0000:80:02.0/0000:82:00.0/host1/port-1:0/expander-1:0/port-1:0:0/end_device-1:0:0/target1:0:0/1:0:0:0/block/sde lrwxrwxrwx 1 root root 0 Jul 25 09:03 sdeg -> ../../../../pci0000:80/0000:80:03.0/0000:83:00.0/host5/port-5:1/expander-5:1/port-5:1:0/end_device-5:1:0/target5:0:45/5:0:45:0/block/sdeg Is there any chance that multipath devices will be supported by PVE in the near future? I'd be willing to test on my non-production system… Regards, Uwe Am 20.07.23 um 14:21 schrieb Uwe Sauter: > Dear all, > > I'd like to use some existing hardware to create a Ceph cluster. Unfortunately, all my external > disks are filtered by the WebUI and by pceceph command which prevents using them as OSD disks. > > My servers are connected to a SAS-JBOD containing 60 SAS-HDDs. The connection is done via 2 > dual-port SAS-HBAs, each port connecting to both of the JBOD controllers. > This means that all my external disks can be reached via four SAS-path and thus I see 240 disks in > /dev. (Actually, I see 244 /dev/sd* devices as there are also 4 internal disks for the OS…) > Using multipathd, each disk with its four entries in /dev is available a fifth time in > /dev/mapper/${WWN} which is a symlink to /dev/dm-${NUMBER}. > > Both /dev/dm-${NUMBER} and /dev/mapper/${WWN} entries are filtered by Proxmox. > > Is there a way to remove this filter or to define my own set of filters? > > Using the /dev/sd* entries would be possible but I would loose the redundancy provided by multipathd. > > Regards, > > Uwe