From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1166C942D1 for ; Tue, 10 Jan 2023 14:30:42 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DE5E0AE13 for ; Tue, 10 Jan 2023 14:30:11 +0100 (CET) Received: from mail.iper.net (mail.iper.net [94.236.32.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 10 Jan 2023 14:30:10 +0100 (CET) Received: from servercsa.csaricerche.com (UnknownHost [82.134.223.122]) by mail.iper.net with SMTP (version=TLS\Tls12 cipher=Aes256 bits=256); Tue, 10 Jan 2023 14:29:46 +0100 Received: from [192.168.70.10] (psala-lx2.csaricerche.com [192.168.70.10]) (Authenticated sender: psala) by servercsa.csaricerche.com (Postfix) with ESMTPSA id 574F520678798 for ; Tue, 10 Jan 2023 14:29:48 +0100 (CET) Message-ID: Date: Tue, 10 Jan 2023 14:29:47 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Content-Language: it, en-US To: pve-user@lists.proxmox.com References: <7606f44a-279c-c797-8206-95ea281245b1@riminilug.it> <1e2cc9a1-5673-33de-fff6-cf8d601d3806@riminilug.it> From: Piviul In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [0.00 / 20.00]; IP_WHITELIST(0.00)[192.168.70.10] X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: IP_WHITELIST X-Rspamd-Server: servercsa.csaricerche.com X-Rspamd-Queue-Id: 574F520678798 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.275 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_SOFTFAIL 0.665 SPF: sender does not match SPF record (softfail) Subject: Re: [PVE-User] ceph X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jan 2023 13:30:42 -0000 On 1/10/23 09:04, Eneko Lacunza via pve-user wrote: > > I think you may have a wrong Ceph network definition in > /etc/pve/ceph.conf, check for "public_network". # cat /etc/pve/ceph.conf [global]      auth_client_required = cephx      auth_cluster_required = cephx      auth_service_required = cephx      cluster_network = 192.168.254.3/24      fsid = 332d7723-d16f-443f-947d-a5ab160e4fac      mon_allow_pool_delete = true      mon_host = 192.168.70.34 192.168.70.30 192.168.70.32      osd_pool_default_min_size = 2      osd_pool_default_size = 3      public_network = 192.168.70.34/20 [client]      keyring = /etc/pve/priv/$cluster.$name.keyring [mds]      keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.pve01]      host = pve01      mds standby for name = pve [mds.pve02]      host = pve02      mds_standby_for_name = pve [mds.pve03]      host = pve03      mds_standby_for_name = pve [mon.pve01]      public_addr = 192.168.70.30 [mon.pve02]      public_addr = 192.168.70.32 [mon.pve03]      public_addr = 192.168.70.34 > Also check what IP are listening on OSD daemons. # netstat -tunlp | grep ceph- tcp        0      0 192.168.254.1:6806      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.70.30:6806      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.254.1:6807      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.70.30:6807      0.0.0.0:* LISTEN      1761/ceph-osd tcp        0      0 192.168.254.1:6808      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.70.30:6808      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.254.1:6809      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.70.30:6809      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.254.1:6810      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.70.30:6810      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.254.1:6811      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.70.30:6811      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.254.1:6812      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.70.30:6812      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.254.1:6813      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.70.30:6813      0.0.0.0:* LISTEN      1762/ceph-osd tcp        0      0 192.168.254.1:6814      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.70.30:6814      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.254.1:6815      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.70.30:6815      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.70.30:6816      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.70.30:6817      0.0.0.0:* LISTEN      68106/ceph-osd tcp        0      0 192.168.70.30:3300      0.0.0.0:* LISTEN      1753/ceph-mon tcp        0      0 192.168.70.30:6789      0.0.0.0:* LISTEN      1753/ceph-mon tcp        0      0 192.168.254.1:6800      0.0.0.0:* LISTEN      1761/ceph-osd tcp        0      0 192.168.70.30:6800      0.0.0.0:* LISTEN      1740/ceph-mds tcp        0      0 192.168.254.1:6801      0.0.0.0:* LISTEN      1761/ceph-osd tcp        0      0 192.168.70.30:6801      0.0.0.0:* LISTEN      1740/ceph-mds tcp        0      0 192.168.254.1:6802      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.70.30:6802      0.0.0.0:* LISTEN      1761/ceph-osd tcp        0      0 192.168.254.1:6803      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.70.30:6803      0.0.0.0:* LISTEN      1761/ceph-osd tcp        0      0 192.168.254.1:6804      0.0.0.0:* LISTEN      1761/ceph-osd tcp        0      0 192.168.70.30:6804      0.0.0.0:* LISTEN      1760/ceph-osd tcp        0      0 192.168.254.1:6805      0.0.0.0:* LISTEN      1761/ceph-osd tcp        0      0 192.168.70.30:6805      0.0.0.0:* LISTEN      1761/ceph-osd I wrong something on CEPH configuration? Piviul