From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <t.lamprecht@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id BC43960982 for <pve-user@lists.proxmox.com>; Thu, 26 Nov 2020 19:56:48 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B9EB615738 for <pve-user@lists.proxmox.com>; Thu, 26 Nov 2020 19:56:48 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id CB5E61572C for <pve-user@lists.proxmox.com>; Thu, 26 Nov 2020 19:56:46 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 7CED845CD3 for <pve-user@lists.proxmox.com>; Thu, 26 Nov 2020 19:56:45 +0100 (CET) To: Proxmox VE user list <pve-user@lists.proxmox.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> From: Thomas Lamprecht <t.lamprecht@proxmox.com> Message-ID: <00c86fc4-6d3a-8a64-db84-27d5b5558617@proxmox.com> Date: Thu, 26 Nov 2020 19:56:44 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Thunderbird/83.0 MIME-Version: 1.0 In-Reply-To: <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.077 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [remote-fs.target, remote-pre.target, remote-fs-pre.target] Subject: Re: [PVE-User] Caution: ceph-mon service does not start after today's updates X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Thu, 26 Nov 2020 18:56:48 -0000 Some news. There are a few things at play, it boils down to two things: * a update of various service orderings in ceph with 14.2.12 (released a = bit ago), they introduced pretty much everywhere a `Before=3Dremote-fs-pre.= target` order enforcement. * rrdcached, a service used by pve-cluster.service (pmxcfs), this has no = native systemd service file, so systemd auto generates one, with an `Before=3D= remote-pre.target` order enforcement which then has ordering for the aforementioned `Before=3Dremote-fs-pre.target` Thus you get the cycle (-> means an after odering, all befores where tran= sformed to after by reversing them (systemd does that too)): =2E> pve-cluster -> rrdcached -> remote-pre -> remote-fs-pre -> ceph-mgr@= -. | = | `------------------------------------------------------------------------= ' We're building a new ceph version with the Before=3Dremote-fs-pre.target = removed, it is bogus for the ceph mgr, mds, mon, .. services as is. As you probably guessed, one can also fix this by adapting rrdcached, and= as a work around you can do so: 1. copy over the generated ephemeral service file from /run to /etc, whic= h has higher priority. # cp /run/systemd/generator.late/rrdcached.service /etc/systemd/system/ 2. Drop the after ordering for remote-fs.target # sed -i '/^After=3Dremote-fs.target/d' /etc/systemd/system/rrdcached.ser= vice 3. reboot=20 A ceph 14.2.15-pve2 package will soon be available, we'll also see if we = can improve the rrdcached situation in the future, it has no fault on its own= naturally, the systemd auto generators heuristic is to blame, but maybe w= e can see if upstream or Debian has interest in adding an hand crafted syst= emd unit file, avoiding auto-generation. Otionally we could maintain it for P= VE, or do like in Proxmox Backup Server - use our own rust based RRD implemen= tation regards, Thomas