From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <Bryan@bryanfields.net>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id 148E4952B7
 for <pve-user@lists.proxmox.com>; Wed, 18 Jan 2023 01:32:25 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id EDCD919269
 for <pve-user@lists.proxmox.com>; Wed, 18 Jan 2023 01:32:24 +0100 (CET)
Received: from morty.keekles.org (Morty.keekles.org [199.47.174.151])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS
 for <pve-user@lists.proxmox.com>; Wed, 18 Jan 2023 01:32:22 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
 by morty.keekles.org (Postfix) with ESMTP id 3A6FC19E0C66
 for <pve-user@lists.proxmox.com>; Wed, 18 Jan 2023 00:32:20 +0000 (UTC)
Received: from morty.keekles.org ([127.0.0.1])
 by localhost (morty.keekles.org [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id IeHXY1ZxXglG for <pve-user@lists.proxmox.com>;
 Wed, 18 Jan 2023 00:32:15 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by morty.keekles.org (Postfix) with ESMTP id CE88C19E1BAB
 for <pve-user@lists.proxmox.com>; Wed, 18 Jan 2023 00:32:15 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.10.3 morty.keekles.org CE88C19E1BAB
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bryanfields.net;
 s=909DCF92-EFE7-11EB-9235-648EB8AF1B81; t=1674001935;
 bh=xA8gzddQjSfJTnajBISfWvPx8LiNgmD7+pJzThXfxWI=;
 h=Message-ID:Date:MIME-Version:To:From;
 b=HAfXqqhv6c6IoIcUpmP4d2fEHFk48mqIVBucpFr/OdRCmv4jXFOPHOxUrMktY/NE0
 wSqmPAycLXvQuZTolppaurMdbB3YD7VXZlG/ldE6NfqRhdL0fj/FwLOX2eXcnACKNo
 G74Z3WMRGVMET8LOxbyAgoIeRDslUNWbE5ynTCnKl1eIYFR0U1pjs9yi0Kq8aVdCk9
 C7Xila6cXQe9teCAJjPfmZa0wZ6VhRem21u1PuIuIhhIljYGvOS3oS/F1NMbZ9QEko
 ir5IhEtQEWBBis5bfd0JqmqTmou+UXAHgvA/4FV0ZQVDw4wjB34csO5iOm/KEFjOxB
 tijX8h2nzqI8Q==
X-Virus-Scanned: amavisd-new at morty.keekles.org
Received: from morty.keekles.org ([127.0.0.1])
 by localhost (morty.keekles.org [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id NmtKLyjYjLm3 for <pve-user@lists.proxmox.com>;
 Wed, 18 Jan 2023 00:32:15 +0000 (UTC)
Received: from [192.168.128.105]
 (static-47-206-239-202.tamp.fl.frontiernet.net [47.206.239.202])
 by morty.keekles.org (Postfix) with ESMTPSA id A885419E0C66
 for <pve-user@lists.proxmox.com>; Wed, 18 Jan 2023 00:32:15 +0000 (UTC)
Message-ID: <525f32af-d46a-1299-9b55-fdd9c6d7f429@bryanfields.net>
Date: Tue, 17 Jan 2023 19:32:15 -0500
MIME-Version: 1.0
User-Agent: Mutt/1.12.0 (2019-05-25)
Content-Language: en-US
To: pve-user@lists.proxmox.com
References: <2635f65d-33fb-5447-a3c1-d5cbab9e04e1@bryanfields.net>
 <mailman.261.1673943754.458.pve-user@lists.proxmox.com>
From: Bryan Fields <Bryan@bryanfields.net>
In-Reply-To: <mailman.261.1673943754.458.pve-user@lists.proxmox.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-SPAM-LEVEL: Spam detection results:  0
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DKIM_SIGNED               0.1 Message has a DKIM or DK signature,
 not necessarily valid
 DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
 DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's
 domain
 DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from
 domain
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Subject: Re: [PVE-User] Debian 11 hard lock issues as VM
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, 
 <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, 
 <mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Wed, 18 Jan 2023 00:32:25 -0000

On 1/17/23 3:22 AM, Eneko Lacunza via pve-user wrote:
> Hi Bryan,
>=20
> We started to upgrade our cluster from PVE 7.2 to 7.3 yesterday.
>=20
> I have enabled the agent in our only VM with Debian 11 running on a
> 7.3-4 node at the moment, and performed 5 full backups in a row, VM
> continues working (no hang).

This is replication, but I believe it's the same.

> You haven't provided details about your setup:
>=20
> - Server (especially CPU model). Debian could be suffering from weird
> BIOS clock issues.

The Hosts are HP DL360's Generation 7.  ZFS Raid2 local storage using 1.6=
 TB=20
SAS SSD's.  The life used indicator is now 6% or 7% on most disks.

There is 192 GB of ram in each server 16384=C2=A0MB=C2=A01600=C2=A0MHz EC=
C ram.

There are dual 3.07 GHz 6 core (12 thread) CPU's.  /proc/cpuinfo is below=
.

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz
stepping	: 2
microcode	: 0x1a
cpu MHz		: 1910.971
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 0
cpu cores	: 6
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pa=
t=20
pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx pdpe1gb rdt=
scp=20
lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc=
=20
cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse=
3=20
cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shad=
ow=20
vnmi flexpriority ept vpid dtherm ida arat
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_1gb flexpriority=
=20
tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swa=
pgs=20
itlb_multihit mmio_unknown
bogomips	: 6134.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

the proxmox config for the VM is here:
agent: 1,fstrim_cloned_disks=3D1
bootdisk: scsi0
cores: 2
cpuunits: 2048
ide2: none,media=3Dcdrom
memory: 8192
name: eyes.tampacoop.net
net0: virtio=3D86:49:26:AA:86:E7,bridge=3Dvmbr199,firewall=3D1
net1: virtio=3DA2:C5:47:85:3E:3B,bridge=3Dvmbr8
numa: 0
onboot: 1
ostype: l26
parent: before_extend
scsi0: local-zfs:vm-102-disk-0,discard=3Don,format=3Draw,iothread=3D1,siz=
e=3D48G,ssd=3D1
scsihw: virtio-scsi-single
smbios1: uuid=3D11ed5a86-3395-49f2-ac80-16804b237a0d
sockets: 1
startup: order=3D1
vmgenid: 6238f0f2-ac90-43e0-b56c-05e1ed1c2431


> - Running kernel on PVE 7.3-4 . Kernel 5.15.x has been quite bad for us=
,
> have you tried kernel 5.13 or 5.19?

I reverted to 4.9.0-19-amd64 #1 SMP Debian 4.9.320-2 (2022-06-30) x86_64=20
GNU/Linux Kernel on the guest OS and it's not locked up once now.  This i=
s=20
running either the 5.2.0 or 7.2.0 agent.

I've moved the VM's across hosts and they have the same problem.

FingerlessGloves mentioned there was the possibility of this being a mari=
adb=20
issue and I can confirm we have the official Maria DB packages installed =
on=20
this server.  10.10.2-MariaDB-1:10.10.2+maria~deb11 is what we're running=
 on=20
the server.

Could this be some interaction of new kernel and new maria db?

--=20
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net