From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 150BA61627 for ; Wed, 26 Jul 2023 10:37:31 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id F34152923 for ; Wed, 26 Jul 2023 10:37:30 +0200 (CEST) Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 26 Jul 2023 10:37:29 +0200 (CEST) Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-3fb4146e8ceso54695545e9.0 for ; Wed, 26 Jul 2023 01:37:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690360643; x=1690965443; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:reply-to:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=BYbyyCyvRV+16hfR49QkBudpXYHTuoqJDloLLUmqhBc=; b=hupvR0BlsqzUbY/d2m+oyL/hYXC930AtaVXheEyDUodp9L/lYzzRFyD/DaTTebH6RS Ok4R2t63sEljzknPu80aBlm+I5wpGesOiQSQ2WvIlnhiQxyQx2EYMAsQcsFsNbVqWxCk sPFvmnsUAzSNMUuZiP6BEXnZKR7VeQ2Kr8ObKfHeQixchoAfaPi7zS8UpQ8fBZMjchLD hgBSckJ4vJckU7God3lEFsQfr3aZequ90RmyfNMAx0Yht4sRMG0MWB2eOIVUH0Eogwzw awB17jSjJmzpNN2IvDPt6pSAB0NmLGmzj6MAOLdSo4z+rHpNwMqaBW4crTuF4MRrrn4M aa3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690360643; x=1690965443; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:reply-to:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=BYbyyCyvRV+16hfR49QkBudpXYHTuoqJDloLLUmqhBc=; b=S/9mM9hDXactv92V1DGlyg1+YJ/uBYm036VDcYcve1ywbjauhyS+Vf3RDwKjYFtTc5 hu8Xq5re969+BeNX3VBc7XQ8bfUDz4pLBPl1H2vx+ALks8k3uM4U0BWj9w3F1nI7QoKM JQgMRsFYs2PzgQOjTCs2hbhdnYYU/OojX/VxC4x5OBlKsCj+cSI3vXR8BEyZPqRlK4ln +jbcZpgAqz+fIzj5CGVTySldLkTnomVNV+uROp34bysbAB87MmpaboS9GZRWS8nmCHTU +EnMYYyKiQF7iBCxXJi0sKfzzEUcpcyrDnZlLpgjiz5yfZ/AEGEfufLmmjiqNJ7o/CaF AH8Q== X-Gm-Message-State: ABy/qLaGUhBOy4AOwV2XaZlldteP67513jT2IRx6NrgAM73f+sHx5QWp SNu6gdJI4/fKDYiwjrkuaLc= X-Google-Smtp-Source: APBJJlGKYryTbTSopv4g/GgyJItQB+Z8R0s6ZhdTA/SsevXcpQKtP36wE7U2HbdhWZqZOScGIUqV/w== X-Received: by 2002:a7b:c4da:0:b0:3fb:d1db:545a with SMTP id g26-20020a7bc4da000000b003fbd1db545amr817209wmk.15.1690360643152; Wed, 26 Jul 2023 01:37:23 -0700 (PDT) Received: from ?IPV6:2001:7c0:201d:13b:5605:dbff:fe76:161c? ([2001:7c0:201d:13b:5605:dbff:fe76:161c]) by smtp.googlemail.com with ESMTPSA id 8-20020a05600c028800b003fb225d414fsm1336663wmk.21.2023.07.26.01.37.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 26 Jul 2023 01:37:22 -0700 (PDT) Message-ID: Date: Wed, 26 Jul 2023 10:37:21 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: uwe.sauter.de@gmail.com Content-Language: de-DE To: Alwin Antreich , Proxmox VE user list References: <337ffcc2-2893-93ad-d7b7-8c665b565a89@gmail.com> <2b5b83bb-c90e-b6dd-4b15-a57414b42542@gmail.com> From: Uwe Sauter In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.054 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [PVE-User] DeviceMapper devices get filtered by Proxmox X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jul 2023 08:37:31 -0000 Good morning Alwin, >> Well, if the documentation is to be trusted, there is multipath support since Octupus. >> My use-case is not hyper-converged virtualization; I simply am using Proxmox due to its good UI and >> integration of Ceph (and because it does not rely on containers to deploy Ceph). > I understand, though cephadm isn't that horrible and there are still other ceph solutions out there. ;) If you have air-gapped systems and no containers in your environment, using cephadm would require a whole lot of effort just to make the containers available. On the other hand Proxmox provides this very nice tool to mirror repositories… just saying. >> I am aware that HDDs will need some amount of flash but I do have a couple of SAS-SSDs at hand that >> I can put into the JBODs. And currently all this is just a proof of concept. > Yeah but the ratio is 4 DB/WAL to 1 SSD, opposed to 12:1 for an NVMe. > To add, you will need 512 GB RAM for OSDs (60 x 8GB) alone and at least a 32C/64T CPU. Probably some 2x 25 GbE NIC too, depending on the use-case. For a PoC SSDs and smaller RAM size should be ok. The servers do have 40 cores and 2x 25GbE so that isn't the problem. And once we see if Ceph fits to the rest of our environment we would invest in properly sized hardware. > Just saying, there are certain expectations with that many disks in one node. > >> Thanks for the link though I have to support the arguments of the forum members that multipath is >> an >> enterprise feature that should be supported by an enterprise-class virtualization solution. > Well, multipath is supported. Just not in combination with Ceph. And PVE is not a storage product (yet). I need to disagree on that one. The WebUI disk overview does not show the multipath devices, only the member disks. Yes, there is a comment stating that a device is a multipath member but no way to select the multipath device itself. So, from a users point of view, PVE does not support multipath, it just recognizes multipath members. And the Create ZFS and Create LVM Volume Group setup pop-ups show neither multipath members nor multipath devices. Thanks Uwe