From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 0727669610 for ; Mon, 18 Jan 2021 17:50:22 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id F053C16AD6 for ; Mon, 18 Jan 2021 17:50:21 +0100 (CET) Received: from nsmtp.uni-koblenz.de (nsmtp.uni-koblenz.de [141.26.64.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 2C5C816ABA for ; Mon, 18 Jan 2021 17:50:20 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by nsmtp.uni-koblenz.de (Postfix) with ESMTP id C481922121A for ; Mon, 18 Jan 2021 17:43:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uni-koblenz.de; h=content-transfer-encoding:content-language:content-type :content-type:mime-version:user-agent:date:date:message-id :subject:subject:from:from:received:received:received; s=mail; t=1610988232; x=1612802633; bh=IvcnZH2fRiNAxy7mLbBbaeCszRkNZwmu 5Db+iQVB2E8=; b=AcLeYPmXuSZgK0csBLyrbjhy78iwNVVty9mGMY6aDvVOVN8p HIGJy9XqJtudrFmSWkpEObkE3OlIrKYTpEsD6i0zgpxt+qhjQiV+cE5zAaHG0Sab WtqCZujxa1jhkcJ+FM4RvWeFStLCU+K3kp531F35xeNuu/X+TUd1BRDbU6c= X-Virus-Scanned: Debian amavisd-new at uni-koblenz.de Received: from nsmtp.uni-koblenz.de ([127.0.0.1]) by localhost (nsmtp.uni-koblenz.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fWpatDpKwaXU for ; Mon, 18 Jan 2021 17:43:52 +0100 (CET) Received: from mailproxy.uni-koblenz.de (mailproxy.uni-koblenz.de [141.26.64.230]) by nsmtp.uni-koblenz.de (Postfix) with ESMTPS for ; Mon, 18 Jan 2021 17:43:52 +0100 (CET) Received: from [IPv6:2001:4c80:50:127::506] (mavpn-vlan1.uni-koblenz.de [IPv6:2001:4c80:50:1::40db]) by mailproxy.uni-koblenz.de (Postfix) with ESMTPSA id 7C2FD101CA2 for ; Mon, 18 Jan 2021 17:43:52 +0100 (CET) To: PVE User List From: Rainer Krienke Message-ID: <7fb2c2c2-70a5-6226-617c-4b28aabc2525@uni-koblenz.de> Date: Mon, 18 Jan 2021 17:43:52 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain JMQ_SPF_NEUTRAL 0.5 SPF set to ?all KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [PVE-User] pve & ceph nautilus: Since today I get an error when trying to remove a VM-template X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 16:50:22 -0000 Hello, since today I have a strange Problem with my pve installation. The storage backend used is a "external" ceph nautilus cluster. All VMs are created from templates as full clones. It seems whenever I delete a VM-template I get an error. Removing VMs (not templates) is OK. Removing a template results in this kind of error messages since today: Removing all snapshots: 0% complete...failed. Could not remove disk 'ceph-pxa:base-174-disk-0', check manually: error during cfs-locked 'storage-ceph-pxa' operation: rbd snap purge 'base-174-disk-0' error: Removing all snapshots: 0% complete...failed. Removing all snapshots: 0% complete...failed. error during cfs-locked 'storage-ceph-pxa' operation: rbd snap purge 'base-174-disk-0' error: Removing all snapshots: 0% complete...failed. TASK OK Until today this worked always fine. R Inside of pve the template is deleted, however on ceph the template file is still there in form of a rbd with a protected snapshot (which seems to be sane to me): ceph$ rbd snap ls pxa-rbd/base-174-disk-0 SNAPID NAME SIZE PROTECTED TIMESTAMP 2992 __base__ 32 GiB yes Mon Jan 18 11:59:49 2021 I unprotected the snapshot, removed it and then removed the base-rbd which worked just fine. So on the ceph side everything looks good, but on the pve side I get the strange error. This started today after I started to create a pve-snapshot of a VM and the "include memory" checkbox was on, allthough I didn't want RAM to be included. So I stopped the snapshot via web-gui. I got an error saying: Qemu Guest Agent is not running - VM 185 qmp command 'guest-ping' failed - got timeout snapshot create failed: starting cleanup TASK ERROR: VM 185 qmp command 'savevm-start' failed - VM snapshot already started Perhaps this has left a lock somewhere, but perhaps it has nothing to do with the problem itself. Does anyone have an idea what might be wrong or how to find out more? Thanks Rainer Here some more infos: root@pxsrv1: dpkg -l|grep pve ii pve-cluster 6.2-1 amd64 "pmxcfs" distributed cluster filesystem for Proxmox Virtual Environment. ii pve-container 3.2-2 all Proxmox VE Container management tool ii pve-docs 6.2-6 all Proxmox VE Documentation ii pve-edk2-firmware 2.20200531-1 all edk2 based firmware modules for virtual machines ii pve-firewall 4.1-3 amd64 Proxmox VE Firewall ii pve-firmware 3.1-3 all Binary firmware code for the pve-kernel ii pve-ha-manager 3.1-1 amd64 Proxmox VE HA Manager ii pve-i18n 2.2-1 all Internationalization support for Proxmox VE ii pve-kernel-5.3 6.1-6 all Latest Proxmox VE Kernel Image ii pve-kernel-5.3.10-1-pve 5.3.10-1 amd64 The Proxmox PVE Kernel Image rc pve-kernel-5.3.18-1-pve 5.3.18-1 amd64 The Proxmox PVE Kernel Image ii pve-kernel-5.3.18-3-pve 5.3.18-3 amd64 The Proxmox PVE Kernel Image ii pve-kernel-5.4 6.2-7 all Latest Proxmox VE Kernel Image rc pve-kernel-5.4.34-1-pve 5.4.34-2 amd64 The Proxmox PVE Kernel Image ii pve-kernel-5.4.44-2-pve 5.4.44-2 amd64 The Proxmox PVE Kernel Image ii pve-kernel-5.4.65-1-pve 5.4.65-1 amd64 The Proxmox PVE Kernel Image ii pve-kernel-helper 6.2-7 all Function for various kernel maintenance tasks. ii pve-lxc-syscalld 0.9.1-1 amd64 PVE LXC syscall daemon ii pve-manager 6.2-12 amd64 Proxmox Virtual Environment Management Tools ii pve-qemu-kvm 5.1.0-3 amd64 Full virtualization on x86 hardware ii pve-xtermjs 4.7.0-2 amd64 Binaries built from the Rust termproxy crate ii smartmontools 7.1-pve2 amd64 control and monitor storage systems using S.M.A.R.T. root@pxsrv1:/etc/pve# pvecm status Cluster information ------------------- Name: pxa Config Version: 5 Transport: knet Secure auth: on Quorum information ------------------ Date: Mon Jan 18 14:02:08 2021 Quorum provider: corosync_votequorum Nodes: 5 Node ID: 0x00000001 Ring ID: 1.327 Quorate: Yes Votequorum information ---------------------- Expected votes: 5 Highest expected: 5 Total votes: 5 Quorum: 3 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 a.b.c.d (local) 0x00000002 1 a.b.c.d 0x00000003 1 a.b.c.d 0x00000004 1 a.b.c.d 0x00000005 1 a.b.c.d root@pxa1:/etc/pve# root@pxa1:/etc/pve# pvecm status Cluster information ------------------- Name: pxa Config Version: 5 Transport: knet Secure auth: on Quorum information ------------------ Date: Mon Jan 18 14:06:08 2021 Quorum provider: corosync_votequorum Nodes: 5 Node ID: 0x00000001 Ring ID: 1.327 Quorate: Yes Votequorum information ---------------------- Expected votes: 5 Highest expected: 5 Total votes: 5 Quorum: 3 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 a.b.c.d 0x00000002 1 a.b.c.d 0x00000003 1 a.b.c.d 0x00000004 1 a.b.c.d 0x00000005 1 a.b.c.d -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312