From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id AFECF76320 for ; Thu, 15 Jul 2021 11:32:04 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9C1871C97D for ; Thu, 15 Jul 2021 11:31:34 +0200 (CEST) Received: from email.corp.tncrew.net.au (email.corp.tncrew.net.au [103.127.162.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 2DDEE1C96E for ; Thu, 15 Jul 2021 11:31:31 +0200 (CEST) Received: from [180.150.12.121] (port=43073 helo=[10.11.12.136]) by email.corp.tncrew.net.au with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m3xiJ-003dAE-94; Thu, 15 Jul 2021 19:31:22 +1000 From: Luke Thompson To: pve-user@lists.proxmox.com, Eneko Lacunza References: Message-ID: <6963c47e-f41f-dc97-c328-f308efc08e9a@tncrew.com.au> Date: Thu, 15 Jul 2021 19:31:21 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - email.corp.tncrew.net.au X-AntiAbuse: Original Domain - lists.proxmox.com X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - tncrew.com.au X-Get-Message-Sender-Via: email.corp.tncrew.net.au: authenticated_id: luke.t@tncrew.com.au X-Authenticated-Sender: email.corp.tncrew.net.au: luke.t@tncrew.com.au X-Source: X-Source-Args: X-Source-Dir: X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [gigabyte.com, proxmox.com, arstechnica.com, binovo.es] Subject: Re: [PVE-User] Proxmox node hang X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jul 2021 09:32:04 -0000 Hi Eneko, Consumer RAM is always a tricky starting point. When under heavy load, the fault rates tend to be quite surprising (hence why ECC is preferable in enterprise/etc settings). Your Gigabyte B450 Aurus M motherboard has many new BIOS iterations available - they're definitely worth reading into, and applying after your own research. https://www.gigabyte.com/Motherboard/B450-AORUS-M-rev-1x/support#support-dl-bios (F60/F61c in 2021 compared to F50 in 2019) Where there's been a hardware fault downstream, we tend to see similar kernel taint flags/states to what you are with P O / P D O / etc. https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html (We've had hardware like OOB cards trigger module flags) Have you run the system through extended memory testing (outside of PVE, which includes memtest at-boot, at least under the ISO)? Your BIOS version is dated 1 month after this article, which would imply that a BIOS update may be beneficial to avoiding the RNG bug. https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd-microcode-bug-destroyed-my-weekend With the kernel taint lines, do you have the procs/calls that marry up to the PIDs listed? What were they doing at the time? From the brief logs you've included, they look varied implying that the problem is likely hardware-based. I'd guess RAM. As it's only faulted once, I'd say a decent course of action would be to test the memory extensively, and go from there. If it can identify a faulting module, then you can remove that DIMM and swap it for a known-good one instead, etc. The testing can take a while, and in our experience it can be worth leaving it to cycle through, esp. with non-ECC. Even though tainted kernels never "lose their taint", if you remove the underlying cause it should clear the state. It'll be good to hear about how you get on with it all. Best of luck with it! Cheers, Luke Thompson Operations Manager luke.t@tncrew.com.au PO Box 111, West Wallsend On 15/7/21 6:40 pm, Eneko Lacunza via pve-user wrote: > Hi all, > > Tonight a node of our 5-node Proxmox 6.4+Ceph cluster has frezeed at > ~6:45. A reset has brought it online later in the morning and is working > well for 2 hours right now. > > HA worked like a charm and Ceph has recovered in some minutes. > > Fantastic success history really, thanks for your excelent work Proxmox > developer and contributors! > > Now for the "post-mortem", I see node's 8 cores "general protection > fault"ing one after another in a minute, with different processes. > > I suspect a memory module or main board fault (Ryzen 3700X 8-core, > 4x32GB non-ECC RAM and gigabyte mainboard, all "consumer" parts, it has > been working well since dec 2019). What do you think? > > Here a shortened syslog (I can provide all 437 lines if necessary): > > --- > > Jul 15 06:45:00 sanmarko systemd[1]: Starting Proxmox VE replication > runner... > > Jul 15 06:45:00 sanmarko systemd[1]: pvesr.service: Succeeded. > > Jul 15 06:45:00 sanmarko systemd[1]: Started Proxmox VE replication runner. > > Jul 15 06:45:01 sanmarko CRON[1913457]: (root) CMD (command -v > debian-sa1 > /dev/null && debian-sa1 1 1) > > Jul 15 06:45:01 sanmarko CRON[1913458]: (root) CMD (if [ -x > /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update > 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then > /etc/munin/plugins/ap > > t update 7200 12 >/dev/null; fi) > > Jul 15 06:45:10 sanmarko kernel: [145747.429110] general protection > fault: 0000 [#1] SMP NOPTI > > Jul 15 06:45:10 sanmarko kernel: [145747.429175] CPU: 11 PID: 1914237 > Comm: ceph Tainted: P           O      5.4.124-1-pve #1 > > Jul 15 06:45:10 sanmarko kernel: [145747.429245] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:10 sanmarko kernel: [145747.429322] RIP: > 0010:kmem_cache_alloc+0x89/0x240 > > Jul 15 06:45:10 sanmarko kernel: [145747.429382] Code: 08 65 4c 03 05 30 > e4 57 5b 49 83 78 10 00 4d 8b 20 0f 84 94 01 00 00 4d 85 e4 0f 84 8b 01 > 00 00 41 8b 47 20 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f 7 > > 0 01 00 00 4c 89 e0 48 0f c9 48 31 cb > > [...] > > Jul 15 06:45:10 sanmarko kernel: [145747.430245] Call Trace: > > Jul 15 06:45:10 sanmarko kernel: [145747.430314]  ? > security_file_alloc+0x29/0x90 > > [...] > > Jul 15 06:45:10 sanmarko kernel: [145747.431244] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > [...] > > Jul 15 06:45:12 sanmarko kernel: [145749.037616] general protection > fault: 0000 [#2] SMP NOPTI > > Jul 15 06:45:12 sanmarko kernel: [145749.037695] CPU: 11 PID: 2433 Comm: > tp_fstore_op Tainted: P      D    O      5.4.124-1-pve #1 > > Jul 15 06:45:12 sanmarko kernel: [145749.037793] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:12 sanmarko kernel: [145749.037898] RIP: > 0010:apparmor_file_free_security+0x22/0x40 > > Jul 15 06:45:12 sanmarko kernel: [145749.037975] Code: 2c ff ff eb a2 0f > 1f 00 0f 1f 44 00 00 48 63 05 28 fb fc 00 48 03 87 c0 00 00 00 74 1a 48 > 8b 78 08 48 85 ff 74 11 55 48 89 e5 ff 0f 0f 88 dc 96 62 00 74 03 > 5d c3 c3 e8 db 55 00 00 5d c3 66 > > [...] > > Jul 15 06:45:12 sanmarko kernel: [145749.038942] Call Trace: > > Jul 15 06:45:12 sanmarko kernel: [145749.039015] > security_file_free+0x27/0x60 > > [...] > > Jul 15 06:45:12 sanmarko kernel: [145749.039441] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > [...] > > Jul 15 06:45:29 sanmarko kernel: [145765.573841] general protection > fault: 0000 [#3] SMP NOPTI > > Jul 15 06:45:29 sanmarko kernel: [145765.573922] CPU: 11 PID: 1733 Comm: > pve-firewall Tainted: P      D    O      5.4.124-1-pve #1 > > Jul 15 06:45:29 sanmarko kernel: [145765.574021] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:29 sanmarko kernel: [145765.574127] RIP: > 0010:kmem_cache_alloc+0x89/0x240 > > Jul 15 06:45:29 sanmarko kernel: [145765.574201] Code: 08 65 4c 03 05 30 > e4 57 5b 49 83 78 10 00 4d 8b 20 0f 84 94 01 00 00 4d 85 e4 0f 84 8b 01 > 00 00 41 8b 47 20 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f 70 01 > 00 00 4c 89 e0 48 0f c9 48 31 cb > > [...] > > Jul 15 06:45:29 sanmarko kernel: [145765.576321] Call Trace: > > Jul 15 06:45:29 sanmarko kernel: [145765.576391]  ? > security_file_alloc+0x29/0x90 > > [...] > > Jul 15 06:45:29 sanmarko kernel: [145765.577258] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > [...] > > Jul 15 06:45:29 sanmarko systemd[1]: pve-firewall.service: Main process > exited, code=killed, status=11/SEGV > > Jul 15 06:45:29 sanmarko systemd[1]: pve-firewall.service: Failed with > result 'signal'. > > Jul 15 06:45:35 sanmarko kernel: [145772.194438] general protection > fault: 0000 [#4] SMP NOPTI > > Jul 15 06:45:35 sanmarko kernel: [145772.194516] CPU: 11 PID: 1776 Comm: > ms_dispatch Tainted: P      D    O      5.4.124-1-pve #1 > > Jul 15 06:45:35 sanmarko kernel: [145772.194614] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:35 sanmarko kernel: [145772.194718] RIP: > 0010:kmem_cache_alloc+0x89/0x240 > > Jul 15 06:45:35 sanmarko kernel: [145772.194792] Code: 08 65 4c 03 05 30 > e4 57 5b 49 83 78 10 00 4d 8b 20 0f 84 94 01 00 00 4d 85 e4 0f 84 8b 01 > 00 00 41 8b 47 20 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f 70 01 > 00 00 4c 89 e0 48 0f c9 48 31 cb > > [...] > > Jul 15 06:45:35 sanmarko kernel: [145772.195750] Call Trace: > > Jul 15 06:45:35 sanmarko kernel: [145772.195819]  ? > security_file_alloc+0x29/0x90 > > [...] > > Jul 15 06:45:35 sanmarko kernel: [145772.197176] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > [...] > > Jul 15 06:45:37 sanmarko kernel: [145774.137506] general protection > fault: 0000 [#5] SMP NOPTI > > Jul 15 06:45:37 sanmarko kernel: [145774.137586] CPU: 11 PID: 2466 Comm: > tp_fstore_op Tainted: P      D    O      5.4.124-1-pve #1 > > Jul 15 06:45:37 sanmarko kernel: [145774.137687] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:37 sanmarko kernel: [145774.137791] RIP: > 0010:kmem_cache_alloc+0x89/0x240 > > Jul 15 06:45:37 sanmarko kernel: [145774.137865] Code: 08 65 4c 03 05 30 > e4 57 5b 49 83 78 10 00 4d 8b 20 0f 84 94 01 00 00 4d 85 e4 0f 84 8b 01 > 00 00 41 8b 47 20 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f 70 01 > 00 00 4c 89 e0 48 0f c9 48 31 cb > > [...] > > Jul 15 06:45:37 sanmarko kernel: [145774.139990] Call Trace: > > Jul 15 06:45:37 sanmarko kernel: [145774.140059]  ? > security_file_alloc+0x29/0x90 > > [...] > > Jul 15 06:45:37 sanmarko kernel: [145774.140991] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > [...] > > Jul 15 06:45:40 sanmarko kernel: [145776.830930] general protection > fault: 0000 [#6] SMP NOPTI > > Jul 15 06:45:40 sanmarko kernel: [145776.831010] CPU: 11 PID: 7234 Comm: > kvm Tainted: P      D    O      5.4.124-1-pve #1 > > Jul 15 06:45:40 sanmarko kernel: [145776.831109] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:40 sanmarko kernel: [145776.831217] RIP: > 0010:kmem_cache_alloc+0x89/0x240 > > Jul 15 06:45:40 sanmarko kernel: [145776.831294] Code: 08 65 4c 03 05 30 > e4 57 5b 49 83 78 10 00 4d 8b 20 0f 84 94 01 00 00 4d 85 e4 0f 84 8b 01 > 00 00 41 8b 47 20 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f 70 01 > 00 00 4c 89 e0 48 0f c9 48 31 cb > > [...] > > Jul 15 06:45:40 sanmarko kernel: [145776.832264] Call Trace: > > Jul 15 06:45:40 sanmarko kernel: [145776.832334]  ? > security_file_alloc+0x29/0x90 > > [...] > > Jul 15 06:45:40 sanmarko kernel: [145776.833336] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > Jul 15 06:45:40 sanmarko pve-ha-lrm[1914439]: starting service vm:149 > > Jul 15 06:45:40 sanmarko pve-ha-lrm[1914439]: starting task > UPID:sanmarko:001D3649:00DE7138:60EFBD74:qmstart:149:root@pam: > > Jul 15 06:45:40 sanmarko pve-ha-lrm[1914441]: start VM 149: > UPID:sanmarko:001D3649:00DE7138:60EFBD74:qmstart:149:root@pam: > > Jul 15 06:45:40 sanmarko systemd[1]: 149.scope: Succeeded. > > Jul 15 06:45:40 sanmarko systemd[1]: Stopped 149.scope. > > Jul 15 06:45:43 sanmarko kernel: [145779.840863] general protection > fault: 0000 [#7] SMP NOPTI > > Jul 15 06:45:43 sanmarko kernel: [145779.840942] CPU: 11 PID: 1740 Comm: > pvestatd Tainted: P      D    O      5.4.124-1-pve #1 > > Jul 15 06:45:43 sanmarko kernel: [145779.842207] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:43 sanmarko kernel: [145779.842310] RIP: > 0010:kmem_cache_alloc+0x89/0x240 > > Jul 15 06:45:43 sanmarko kernel: [145779.842383] Code: 08 65 4c 03 05 30 > e4 57 5b 49 83 78 10 00 4d 8b 20 0f 84 94 01 00 00 4d 85 e4 0f 84 8b 01 > 00 00 41 8b 47 20 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f 70 01 > 00 00 4c 89 e0 48 0f c9 48 31 cb > > [...] > > Jul 15 06:45:43 sanmarko kernel: [145779.843337] Call Trace: > > Jul 15 06:45:43 sanmarko kernel: [145779.843406]  ? > security_file_alloc+0x29/0x90 > > [...] > > Jul 15 06:45:43 sanmarko kernel: [145779.844545] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > [...] > > Jul 15 06:45:43 sanmarko systemd[1]: pvestatd.service: Main process > exited, code=killed, status=11/SEGV > > Jul 15 06:45:43 sanmarko systemd[1]: pvestatd.service: Failed with > result 'signal'. > > Jul 15 06:45:45 sanmarko pve-ha-lrm[1914439]: Task > 'UPID:sanmarko:001D3649:00DE7138:60EFBD74:qmstart:149:root@pam:' still > active, waiting > > Jul 15 06:45:45 sanmarko pve-ha-lrm[1914441]: timeout waiting on systemd > > Jul 15 06:45:45 sanmarko pve-ha-lrm[1914439]: end task > UPID:sanmarko:001D3649:00DE7138:60EFBD74:qmstart:149:root@pam: timeout > waiting on systemd > > Jul 15 06:45:45 sanmarko pve-ha-lrm[1914439]: unable to start service vm:149 > > Jul 15 06:45:50 sanmarko pve-ha-lrm[1804]: restart policy: retry number > 1 for service 'vm:149' > > Jul 15 06:45:56 sanmarko kernel: [145792.695695] general protection > fault: 0000 [#8] SMP NOPTI > > Jul 15 06:45:56 sanmarko kernel: [145792.695777] CPU: 11 PID: 1783 Comm: > pve-ha-crm Tainted: P      D    O      5.4.124-1-pve #1 > > Jul 15 06:45:56 sanmarko kernel: [145792.695876] Hardware name: Gigabyte > Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F50 11/27/2019 > > Jul 15 06:45:56 sanmarko kernel: [145792.695980] RIP: > 0010:kmem_cache_alloc+0x89/0x240 > > Jul 15 06:45:56 sanmarko kernel: [145792.696054] Code: 08 65 4c 03 05 30 > e4 57 5b 49 83 78 10 00 4d 8b 20 0f 84 94 01 00 00 4d 85 e4 0f 84 8b 01 > 00 00 41 8b 47 20 49 8b 3f 4c 01 e0 <48> 8b 18 48 89 c1 49 33 9f 70 01 > 00 00 4c 89 e0 48 0f c9 48 31 cb > > [...] > > Jul 15 06:45:56 sanmarko kernel: [145792.697012] Call Trace: > > Jul 15 06:45:56 sanmarko kernel: [145792.697081]  ? > security_file_alloc+0x29/0x90 > > [...] > > Jul 15 06:45:56 sanmarko kernel: [145792.698225] > entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > [...] > > Jul 15 06:45:56 sanmarko watchdog-mux[895]: client did not stop watchdog > - disable watchdog updates > > Jul 15 06:45:56 sanmarko systemd[1]: pve-ha-crm.service: Main process > exited, code=killed, status=11/SEGV > > Jul 15 06:45:56 sanmarko systemd[1]: pve-ha-crm.service: Failed with > result 'signal'. > > Jul 15 06:45:56 sanmarko kernel: [145792.701730] FS: > 00007fae7b4141c0(0000) GS:ffff96b69eac0000(0000) knlGS:0000000000000000 > > Jul 15 06:45:56 sanmarko kernel: [145792.701826] CS:  0010 DS: 0000 ES: > 0000 CR0: 0000000080050033 > > Jul 15 06:45:56 sanmarko kernel: [145792.701902] CR2: 00007fa6bbf73008 > CR3: 0000001f29900000 CR4: 0000000000340ee0 > > [... no more logs until reset ...] > > > # pveversion -v > > proxmox-ve: 6.4-1 (running kernel: 5.4.124-1-pve) > pve-manager: 6.4-13 (running version: 6.4-13/9f411e79) > pve-kernel-5.4: 6.4-4 > pve-kernel-helper: 6.4-4 > pve-kernel-5.3: 6.1-6 > pve-kernel-5.4.124-1-pve: 5.4.124-1 > pve-kernel-5.4.119-1-pve: 5.4.119-1 > pve-kernel-5.3.18-3-pve: 5.3.18-3 > ceph: 15.2.13-pve1~bpo10 > ceph-fuse: 15.2.13-pve1~bpo10 > corosync: 3.1.2-pve1 > criu: 3.11-3 > glusterfs-client: 5.5-3 > ifupdown: residual config > ifupdown2: 3.0.0-1+pve4~bpo10 > libjs-extjs: 6.0.1-10 > libknet1: 1.20-pve1 > libproxmox-acme-perl: 1.1.0 > libproxmox-backup-qemu0: 1.1.0-1 > libpve-access-control: 6.4-3 > libpve-apiclient-perl: 3.1-3 > libpve-common-perl: 6.4-3 > libpve-guest-common-perl: 3.1-5 > libpve-http-server-perl: 3.2-3 > libpve-storage-perl: 6.4-1 > libqb0: 1.0.5-1 > libspice-server1: 0.14.2-4~pve6+1 > lvm2: 2.03.02-pve4 > lxc-pve: 4.0.6-2 > lxcfs: 4.0.6-pve1 > novnc-pve: 1.1.0-1 > proxmox-backup-client: 1.1.10-1 > proxmox-mini-journalreader: 1.1-1 > proxmox-widget-toolkit: 2.6-1 > pve-cluster: 6.4-1 > pve-container: 3.3-6 > pve-docs: 6.4-2 > pve-edk2-firmware: 2.20200531-1 > pve-firewall: 4.1-4 > pve-firmware: 3.2-4 > pve-ha-manager: 3.1-1 > pve-i18n: 2.3-1 > pve-qemu-kvm: 5.2.0-6 > pve-xtermjs: 4.7.0-3 > qemu-server: 6.4-2 > smartmontools: 7.2-pve2 > spiceterm: 3.1-1 > vncterm: 1.6-2 > zfsutils-linux: 2.0.4-pve1 > > > Thanks a lot > > > Eneko Lacunza > Zuzendari teknikoa | Director técnico > Binovo IT Human Project > > Tel. +34 943 569 206 |https://www.binovo.es > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun > > https://www.youtube.com/user/CANALBINOVO > https://www.linkedin.com/company/37269706/ > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user