From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6ED68711A4 for ; Wed, 8 Jun 2022 12:13:45 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 62BA467D4 for ; Wed, 8 Jun 2022 12:13:45 +0200 (CEST) Received: from mailhost2.inet.dkfz-heidelberg.de (mailhost2.inet.dkfz-heidelberg.de [193.174.53.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 822BF67CB for ; Wed, 8 Jun 2022 12:13:44 +0200 (CEST) Received: from mx-ext.inet.dkfz-heidelberg.de (mx-ext.inet.dkfz-heidelberg.de [192.54.49.101]) by mailhost2.inet.dkfz-heidelberg.de (8.14.7/8.14.7) with ESMTP id 258A8N9Q008574 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 8 Jun 2022 12:08:23 +0200 X-Virus-Scanned-DKFZ: amavisd-new at dkfz-heidelberg.de Received: from [172.22.165.134] (addr16.inet.dkfz-heidelberg.de [193.174.55.143]) (authenticated bits=0) by mx-ext.inet.dkfz-heidelberg.de (8.14.7/8.14.7/smtpin) with ESMTP id 258A8IsH011824 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 8 Jun 2022 12:08:21 +0200 DKIM-Filter: OpenDKIM Filter v2.11.0 mx-ext.inet.dkfz-heidelberg.de 258A8IsH011824 Message-ID: <19fec87f-3c95-b03c-5a50-de649771cc3a@dkfz-heidelberg.de> Date: Wed, 8 Jun 2022 12:08:18 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 From: Frank Thommen To: Proxmox VE user list Organization: DKFZ Heidelberg, Omics IT and Data Management Core Facility (ODCF) Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (mx-ext.inet.dkfz-heidelberg.de [192.54.49.101]); Wed, 08 Jun 2022 12:08:21 +0200 (CEST) X-Spam-Status: No, score=-100.0 required=5.0 tests=ALL_TRUSTED, T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mx-ext.inet.dkfz-heidelberg.de X-SPAM-LEVEL: Spam detection results: 0 AWL -0.567 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [PVE-User] Service "systemd-tmpfiles-setup" fails since upgrade to PVE 7.x X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2022 10:13:45 -0000 Dear all, since the upgrade of our PVE environment from 6.x to 7.2-4 last week, our monitoring (CheckMK) registers, that the service "systemd-tmpfiles-setup" fails in all containers (not in the VMs, though). The containers have been created from the provided CentOS 7.9 templates and are fully updated. The containers have all been created under PVE 5.x and 6.x. I'm not familiar with "systemd-tmpfiles-setup" and I could not relate similar failure reports found in the net to our situation. The status that we get is: -------------------------------------------------------------- [root@odcf-vm119 ~]# systemctl status systemd-tmpfiles-setup.service ● systemd-tmpfiles-setup.service - Create Volatile Files and Directories Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2022-06-07 12:28:04 CEST; 23h ago Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Process: 55 ExecStart=/usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev (code=exited, status=1/FAILURE) Main PID: 55 (code=exited, status=1/FAILURE) Jun 07 12:28:03 odcf-vm119 systemd[1]: Starting Create Volatile Files and Directories... Jun 07 12:28:03 odcf-vm119 systemd-tmpfiles[55]: Failed to create directory or subvolume "/var/lib/machines": Operation not permitted Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create directory or subvolume "/tmp": Operation not permitted Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create directory or subvolume "/var/tmp": Operation not permitted Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service: main process exited, code=exited, status=1/FAILURE Jun 07 12:28:04 odcf-vm119 systemd[1]: Failed to start Create Volatile Files and Directories. Jun 07 12:28:04 odcf-vm119 systemd[1]: Unit systemd-tmpfiles-setup.service entered failed state. Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service failed. [root@odcf-vm119 ~]# -------------------------------------------------------------- journalctl tells me (manual extract from `journalctl -t systemd`): -------------------------------------------------------------- Jun 07 12:28:03 odcf-vm119 systemd[1]: Starting Create Volatile Files and Directories... Jun 07 12:28:04 odcf-vm119 systemd[1]: Started Load/Save Random Seed. Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service: main process exited, code=exited, status=1/FAILURE Jun 07 12:28:04 odcf-vm119 systemd[1]: Failed to start Create Volatile Files and Directories. Jun 07 12:28:04 odcf-vm119 systemd[1]: Unit systemd-tmpfiles-setup.service entered failed state. Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service failed. -------------------------------------------------------------- and (`journalctl -t systemd-tmpfiles`): -------------------------------------------------------------- -- Logs begin at Tue 2022-06-07 12:28:03 CEST, end at Wed 2022-06-08 12:00:47 CEST. -- Jun 07 12:28:03 odcf-vm119 systemd-tmpfiles[55]: Failed to create directory or subvolume "/var/lib/machines": Operation not permitted Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create directory or subvolume "/tmp": Operation not permitted Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create directory or subvolume "/var/tmp": Operation not permitted -------------------------------------------------------------- Can anyone tell, what the problem could be or in which direction we should try to search for the problem? I might also just migrate all containers to VMs, if that is a good or recommended solution to get rid of this problem. I currently don't see the benefit of containers over VMs anyway at this moment :-) Cheers and thanks in advance Frank