From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <uwe.sauter.de@gmail.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id C3F72FDBD
for <pve-user@pve.proxmox.com>; Tue, 25 Jul 2023 09:24:33 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id ACF13166EF
for <pve-user@pve.proxmox.com>; Tue, 25 Jul 2023 09:24:33 +0200 (CEST)
Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com
[IPv6:2a00:1450:4864:20::52b])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-user@pve.proxmox.com>; Tue, 25 Jul 2023 09:24:33 +0200 (CEST)
Received: by mail-ed1-x52b.google.com with SMTP id
4fb4d7f45d1cf-51ff0e3d8c1so7402397a12.0
for <pve-user@pve.proxmox.com>; Tue, 25 Jul 2023 00:24:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20221208; t=1690269866; x=1690874666;
h=content-transfer-encoding:in-reply-to:references:to:from
:content-language:subject:reply-to:user-agent:mime-version:date
:message-id:from:to:cc:subject:date:message-id:reply-to;
bh=NoCrBaj3IyC9ZWgWmOnG4mQt05/5oeM/0ulnA6p/9WE=;
b=htjMcng0TapTN3PTQ6t4mc/bQdwdPNruuDlIGOUnwjFYyNeA8UJD09kf6qlwR1A5JS
FLGPOZv+bD65oEIiLawWgeqkdT54V5e19xDv2O/1qWw0TbcUiXF/b7X0aaC4S3KSiQ67
9EMWNp7MnTg1/PJ0MMEpZ7p2CnriWpbLqkVk+7/T+1wnSuI9I37O7tYEdkg2IP0IQGtq
0dUBbIlvro3NmaXxEod1+2Ils0BmhHn/xblgf4CLs/kxz8k6mkOgAsgbXt+R2WGWd+rp
irj964HuuNtWNMq5t+ONvBlkDhRiTVPBVGd55fvTc982/IrHTCYxf0P3G6MWhAnyRxEB
3VKw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20221208; t=1690269866; x=1690874666;
h=content-transfer-encoding:in-reply-to:references:to:from
:content-language:subject:reply-to:user-agent:mime-version:date
:message-id:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=NoCrBaj3IyC9ZWgWmOnG4mQt05/5oeM/0ulnA6p/9WE=;
b=ZnxxMBt+CKOmqyu14JDqKYRNpboFBnME7XZyjkZZ5KaRTspv826KS3DkZiwd0R0bx/
WfEaRgoZWtyUKg/R3O1VdpDJ38ynEBm+50iz4MY9Ttg9bv4sK7l0w2prEQZlYwWA/Ti8
OxwAK6BcOkQMfb8WZ9HZJfM+rTd01RHWJeoePnTtX85iOSHA2hlb1TD9bFapnv+cBRYp
XG4M5eeJFQfQyUG48UkQVyO0+nyU6Tys2pTgvfzpCH2NEjEsJxjizbcYmlzLOWPDc5y/
E0RBKEdMxS3FTv4Ke6szbN6coTAeG6af3foKYHItmREeBUn0XpZpk1L3sOsqYzySYRze
yDeA==
X-Gm-Message-State: ABy/qLaL/siR9Tbo/uE+sBuI3OfY/9YH7m8tdsYqj+B3u3kIIW2zw3Dk
cQfGD3i8TOtP694jyf0CDvYxu+aFKDA=
X-Google-Smtp-Source: APBJJlHLJVGuCEsobmwMsiKnA+TvHoiaIZ4hkxLzxAHrIHpdHZA7+pMDCLbf441ZQjo3WnNOBlqi/w==
X-Received: by 2002:aa7:d1d5:0:b0:522:55bf:21af with SMTP id
g21-20020aa7d1d5000000b0052255bf21afmr413017edp.7.1690269866465;
Tue, 25 Jul 2023 00:24:26 -0700 (PDT)
Received: from ?IPV6:2a02:8070:a280:2d80:5605:dbff:fe76:161d?
([2a02:8070:a280:2d80:5605:dbff:fe76:161d])
by smtp.googlemail.com with ESMTPSA id
v8-20020aa7d808000000b00521d1c34b23sm7137250edq.83.2023.07.25.00.24.25
for <pve-user@pve.proxmox.com>
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Tue, 25 Jul 2023 00:24:25 -0700 (PDT)
Message-ID: <d94e484f-7504-fb07-80a1-13fd52518594@gmail.com>
Date: Tue, 25 Jul 2023 09:24:22 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Reply-To: uwe.sauter.de@gmail.com
Content-Language: de-DE
From: Uwe Sauter <uwe.sauter.de@gmail.com>
To: PVE User List <pve-user@pve.proxmox.com>
References: <2b5b83bb-c90e-b6dd-4b15-a57414b42542@gmail.com>
In-Reply-To: <2b5b83bb-c90e-b6dd-4b15-a57414b42542@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.160 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_SIGNED 0.1 Message has a DKIM or DK signature,
not necessarily valid
DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's
domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from
domain DMARC_PASS -0.1 DMARC pass policy
FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider
RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/,
no trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
T_SCC_BODY_TEXT_LINE -0.01 -
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [diskmanage.pm]
Subject: Re: [PVE-User] DeviceMapper devices get filtered by Proxmox
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 25 Jul 2023 07:24:33 -0000
So, I've been looking further into this and indeed, there seem to be very strict filters regarding
the block device names that Proxmox allows to be used.
/usr/share/perl5/PVE/Diskmanage.pm
512 # whitelisting following devices
513 # - hdX ide block device
514 # - sdX scsi/sata block device
515 # - vdX virtIO block device
516 # - xvdX: xen virtual block device
517 # - nvmeXnY: nvme devices
518 # - cciss!cXnY cciss devices
519 print Dumper($dev);
520 return if $dev !~ m/^(h|s|x?v)d[a-z]+$/ &&
521 $dev !~ m/^nvme\d+n\d+$/ &&
522 $dev !~ m/^cciss\!c\d+d\d+$/;
I don't understand all the consequences of allowing ALL ^dm-\d+$ devices but with proper filtering
it should be possible to allow multipath devices (and given that there might be udev rules that
create additinal symlinks below /dev, each device's name should be resolved to its canonical name
before checking).
To give an example I have sdc as an internal disk that contains a Bluestore OSD:
ls -l /dev/mapper/ | grep dm-0
lrwxrwxrwx 1 root root 7 Jul 22 23:23
ceph--aa6b72f7--f185--44b4--9922--6ae4e6278d10-osd--block--b0a60266--90c0--444f--ae12--328ecfebd87d
-> ../dm-0
Devices sde, sdaw, sdco, sdeg are the same SAS disk and form multipath device dm-2:
# ls -la /dev/mapper/ | grep dm-2$
lrwxrwxrwx 1 root root 7 Jul 22 23:23 35000cca26a7402e4 -> ../dm-2
# multipath -ll | grep -A6 'dm-2 '
35000cca26a7402e4 dm-2 HGST,HUH721010AL5200
size=9.1T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 1:0:0:0 sde 8:64 active ready running
|- 1:0:45:0 sdaw 67:0 active ready running
|- 5:0:0:0 sdco 69:192 active ready running
`- 5:0:45:0 sdeg 128:128 active ready running
Both dm-0 and dm-2 are device mapper devices but it is possible to get their type:
# dmsetup status /dev/dm-0
0 3750739968 linear
# dmsetup status /dev/dm-2
0 19532873728 multipath 2 0 0 0 1 1 A 0 4 2 8:64 A 0 0 1 67:0 A 0 0 1 69:192 A 0 0 1 128:128 A 0 0 1
Another way, possibly less exact, to filter is if /sys/block/dm-X/slaves/ contains more than one entry.
# ls -l /sys/block/dm-0/slaves
total 0
lrwxrwxrwx 1 root root 0 Jul 25 09:01 sdc ->
../../../../pci0000:00/0000:00:11.4/ata3/host3/target3:0:0/3:0:0:0/block/sdc
# ls -l /sys/block/dm-2/slaves
total 0
lrwxrwxrwx 1 root root 0 Jul 25 09:03 sdaw ->
../../../../pci0000:80/0000:80:02.0/0000:82:00.0/host1/port-1:1/expander-1:1/port-1:1:0/end_device-1:1:0/target1:0:45/1:0:45:0/block/sdaw
lrwxrwxrwx 1 root root 0 Jul 25 09:03 sdco ->
../../../../pci0000:80/0000:80:03.0/0000:83:00.0/host5/port-5:0/expander-5:0/port-5:0:0/end_device-5:0:0/target5:0:0/5:0:0:0/block/sdco
lrwxrwxrwx 1 root root 0 Jul 25 09:03 sde ->
../../../../pci0000:80/0000:80:02.0/0000:82:00.0/host1/port-1:0/expander-1:0/port-1:0:0/end_device-1:0:0/target1:0:0/1:0:0:0/block/sde
lrwxrwxrwx 1 root root 0 Jul 25 09:03 sdeg ->
../../../../pci0000:80/0000:80:03.0/0000:83:00.0/host5/port-5:1/expander-5:1/port-5:1:0/end_device-5:1:0/target5:0:45/5:0:45:0/block/sdeg
Is there any chance that multipath devices will be supported by PVE in the near future?
I'd be willing to test on my non-production system…
Regards,
Uwe
Am 20.07.23 um 14:21 schrieb Uwe Sauter:
> Dear all,
>
> I'd like to use some existing hardware to create a Ceph cluster. Unfortunately, all my external
> disks are filtered by the WebUI and by pceceph command which prevents using them as OSD disks.
>
> My servers are connected to a SAS-JBOD containing 60 SAS-HDDs. The connection is done via 2
> dual-port SAS-HBAs, each port connecting to both of the JBOD controllers.
> This means that all my external disks can be reached via four SAS-path and thus I see 240 disks in
> /dev. (Actually, I see 244 /dev/sd* devices as there are also 4 internal disks for the OS…)
> Using multipathd, each disk with its four entries in /dev is available a fifth time in
> /dev/mapper/${WWN} which is a symlink to /dev/dm-${NUMBER}.
>
> Both /dev/dm-${NUMBER} and /dev/mapper/${WWN} entries are filtered by Proxmox.
>
> Is there a way to remove this filter or to define my own set of filters?
>
> Using the /dev/sd* entries would be possible but I would loose the redundancy provided by multipathd.
>
> Regards,
>
> Uwe