From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 08E9A9C107 for ; Tue, 30 May 2023 18:52:22 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DF9FA3227F for ; Tue, 30 May 2023 18:52:21 +0200 (CEST) Received: from mout.web.de (mout.web.de [212.227.17.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 30 May 2023 18:52:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1685465540; i=devzero@web.de; bh=cHmMQjJHB6Du51Yl3ir0IOCJHjSW837HS+L4v6blV5M=; h=X-UI-Sender-Class:Date:Subject:To:References:From:In-Reply-To; b=lyPhsyYaQzhA2HcVzZMdlXhQXByTRZ2A8/qNOEB+8IX5XgypAJ5erIVb9ebqMF+B+ 2fKKkbHAiquzQQ/XFaQeBBPWoQby7RVWuZpicLQU7ogrwk/MEVoamoG/jIpo67X4wy xye+XJ5xT41LgdA1gaUFtxjANpDDFmNlD4CQV8at4/Mebz+ywf8QVwsh/DOfLt9RNw PlSoGNMgKCLxSQYPa+X6dEIhbJdtJjCTBPJA7sXbridysMkD3/Q7esPlYdWbRJ2DMa I3fDhRZ2uu3EO0iW+k6Wq4ydp4RJX/9rMRQiARWSrNWFvKJlr5XzF8Ky7iWDHoBtpC bfxrRijmGIGng== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Received: from [172.20.35.164] ([37.24.118.138]) by smtp.web.de (mrweb105 [213.165.67.124]) with ESMTPSA (Nemesis) id 1N79N8-1q9ivg39KO-017YIE; Tue, 30 May 2023 18:46:52 +0200 Message-ID: Date: Tue, 30 May 2023 18:46:51 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.11.1 To: Christian Schoepplein , pve-user@lists.proxmox.com References: From: Roland In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:oLbLeKkjYQn8tVZNsroZSGw3kM33XwAjkdgBBMxW5irEBLUeXY0 gFjes3TapiA78/AEtCfEucfct9xcnVy5J3MXw2/wxuuGQSsuV/Dv/JMSWhrnoDITKy6lMxj 5DK7hmuZw1WjSkmiXZxNZOweBRQJGgVgMXKCX5Q6TUnl85Uz2a1o6aS3lsZwRKKa+24Y4qD 4WyGZYxjqNyuJ2mw0QxMw== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:RMIRn9whuAI=;qEKsLl7PryCMs2iyB6LBtNHx5Px uhJSSi/C7kpWv/zgeBwbm33zJ14JCura3TXeOYQIYflwusQwO7XzD+R4TxY30hKODkVKAGUiv 681wKYY2EjgDFgSjAytAI59tILvgEJ7bUEeOmC/c+xpK+T51tPLeqBxpXgw5Sb6waR80XBoN+ zdy6y/u1pucyLfSws/W7o+GzxUfa99QUvStRnf318g3BB6sJ/QcS1h2WrqLLsE2VxjQpcEmlh +5d6OEfrN6Ba5shSrJdTSG6z6HZo8sZne2nmhvVzHsOds/FOaVIPsVjlhPX4n0nHKXpu03ziZ V1u0hjN0siXk5C6ZvN1mW8mxZZEPyAs84aDxlc79WoJbHLqSbumiQXYEytrDhkNUtazmX6Fc7 XMAuzkOWEh6reWGdm2QmBjzoh5LYP1VOjrp2DKeATR8sAirgaWnguZFsOtq9ZvfG+sEpO6Ak5 WrCWxKtyKYTsnon4Gvqjs6kfAKXvWa8CDv+0NYyR0UgStiLqGnvrcWPgRyWHUmr4d621GeLWN kpaNspHXwj0n6UkjDbeBwbWBl1jB/lQzR58WFkkn+3+D/Ais1akctlzm+S+BqrUh51iwNt1e5 C7nq3NutBEtaGBxkiTLbbrnLiVHtY6bifIjpcTTfssQvXU6xDROIq/jBDOmG4wJTckF5iKpAq iU0Ad3P8sRfTB8yq9fpKHs45eYymSpVA8HSlgt3tkWZX5g5mZbZpimVf/oGU9GqKm0Xu2pSIV YksfILprD1PaX5SlJKeBBdSnvs5Qx99jG2bX+x5j3/Gnh/9uNnqnBaOYHCwMEogvLMt9XEQav FprnsT/OP9sNDMJrasf4N42BXItAtblYoAwPdPqjtq4puN3+kiIyCbr1YuXCW+osO+F0ozSE4 EnYLTJmoczmOArWx7AGBlpELCyqN478oaO9+lNFs4cnZNcAqe3re2iwiOjUrAh+6R7Cd6FcNT 7gJe6EefcM2qKu6kuaSiNhi4z3A= X-SPAM-LEVEL: Spam detection results: 0 AWL 0.158 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider NICE_REPLY_A -0.091 Looks like a legit reply (A) RCVD_IN_MSPIKE_H3 0.001 Good reputation (+3) RCVD_IN_MSPIKE_WL 0.001 Mailspike good senders SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [PVE-User] Proxmox and glusterfs: VMs get corupted X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 May 2023 16:52:22 -0000 if /mnt/pve/gfs_vms is a writeable path from inside pve host, did you chec= k if there is also corruption when reading/writing large files there and compare with md= 5sum after copy ? furthermore, i remember there was a gluster/qcow2 issue with aio=3Dnative = some years ago, could you retry with aio=3Dthreads for the virtual disks ? regards roland Am 30.05.23 um 18:32 schrieb Christian Schoepplein: > Hi, > > we are testing the current proxmox version with a glusterfs storage back= end > and have a strange issue with file getting corupted inside the virtual > machines. For what reason ever from one moment to another binaries can n= ot > longer be executed, scripts are damaged and so on. In the logs I get err= ors > like this: > > May 30 11:22:36 ns1 dockerd[1234]: time=3D"2023-05-30T11:22:36.874765091= +02:00" level=3Dwarning msg=3D"Running modprobe bridge br_netfilter failed= with message: modprobe: ERROR: could not insert 'bridge': Exec format err= or\nmodprobe: ERROR: could not insert 'br_netfilter': Exec format error\ni= nsmod /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko \ninsmod /lib/m= odules/5.15.0-72-generic/kernel/net/802/stp.ko \n, error: exit status 1" > > On such a broken system a file brings the following: > > root@ns1:~# file /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko > /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko: data > root@ns1:~# > > On a normal system it looks like this: > > root@gluster1:~# file /lib/modules/5.15.0-72-generic/kernel/net/802/stp.= ko > /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko: ELF 64-bit LSB > relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=3D1084f7cfcffbd4c60= 7724fba287c0ea7fc5775 > root@gluster1:~# > > there are not only kernel modules afected. I saw the same behaviour for > scripts, icinga check modules, the sendmail binary and so on, I think it= is > totaly random :-(. > > We have the problems with newly installed VMs, VMs cloned from a templat= e > create on our proxmox host and with VMs which we used before with libvir= td > and migrated to our new proxmox machine. So IMHO it can not be related t= o > the way we create new virtual machines... > > We are using the following software: > > root@proxmox1:~# pveversion -v > proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve) > pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a) > pve-kernel-5.15: 7.4-1 > pve-kernel-5.15.104-1-pve: 5.15.104-2 > pve-kernel-5.15.102-1-pve: 5.15.102-1 > ceph-fuse: 15.2.17-pve1 > corosync: 3.1.7-pve1 > criu: 3.15-1+pve-1 > glusterfs-client: 9.2-1 > ifupdown2: 3.1.0-1+pmx3 > ksm-control-daemon: 1.4-1 > libjs-extjs: 7.0.0-1 > libknet1: 1.24-pve2 > libproxmox-acme-perl: 1.4.4 > libproxmox-backup-qemu0: 1.3.1-1 > libproxmox-rs-perl: 0.2.1 > libpve-access-control: 7.4-2 > libpve-apiclient-perl: 3.2-1 > libpve-common-perl: 7.3-4 > libpve-guest-common-perl: 4.2-4 > libpve-http-server-perl: 4.2-3 > libpve-rs-perl: 0.7.5 > libpve-storage-perl: 7.4-2 > libspice-server1: 0.14.3-2.1 > lvm2: 2.03.11-2.1 > lxc-pve: 5.0.2-2 > lxcfs: 5.0.3-pve1 > novnc-pve: 1.4.0-1 > proxmox-backup-client: 2.4.1-1 > proxmox-backup-file-restore: 2.4.1-1 > proxmox-kernel-helper: 7.4-1 > proxmox-mail-forward: 0.1.1-1 > proxmox-mini-journalreader: 1.3-1 > proxmox-widget-toolkit: 3.6.5 > pve-cluster: 7.3-3 > pve-container: 4.4-3 > pve-docs: 7.4-2 > pve-edk2-firmware: 3.20230228-2 > pve-firewall: 4.3-1 > pve-firmware: 3.6-4 > pve-ha-manager: 3.6.0 > pve-i18n: 2.12-1 > pve-qemu-kvm: 7.2.0-8 > pve-xtermjs: 4.16.0-1 > qemu-server: 7.4-3 > smartmontools: 7.2-pve3 > spiceterm: 3.2-2 > swtpm: 0.8.0~bpo11+3 > vncterm: 1.7-1 > zfsutils-linux: 2.1.9-pve1 > root@proxmox1:~# > > root@proxmox1:~# cat /etc/pve/storage.cfg > dir: local > path /var/lib/vz > content rootdir,iso,images,vztmpl,backup,snippets > > zfspool: local-zfs > pool rpool/data > content images,rootdir > sparse 1 > > glusterfs: gfs_vms > path /mnt/pve/gfs_vms > volume gfs_vms > content images > prune-backups keep-all=3D1 > server gluster1.linova.de > server2 gluster2.linova.de > > root@proxmox1:~# > > The config of a typical VM looks like this: > > root@proxmox1:~# cat /etc/pve/qemu-server/101.conf > #ns1 > agent: enabled=3D1,fstrim_cloned_disks=3D1 > boot: c > bootdisk: scsi0 > cicustom: user=3Dlocal:snippets/user-data > cores: 1 > hotplug: disk,network,usb > ide2: gfs_vms:101/vm-101-cloudinit.qcow2,media=3Dcdrom,size=3D4M > ipconfig0: ip=3D10.200.32.9/22,gw=3D10.200.32.1 > kvm: 1 > machine: q35 > memory: 2048 > meta: creation-qemu=3D7.2.0,ctime=3D1683718002 > name: ns1 > nameserver: 10.200.0.5 > net0: virtio=3D1A:61:75:25:C6:30,bridge=3Dvmbr0 > numa: 1 > ostype: l26 > scsi0: gfs_vms:101/vm-101-disk-0.qcow2,discard=3Don,size=3D10444M > scsihw: virtio-scsi-pci > searchdomain: linova.de > serial0: socket > smbios1: uuid=3De2f503fe-4a66-4085-86c0-bb692add6b7a > sockets: 1 > vmgenid: 3be6ec9d-7cfd-47c0-9f86-23c2e3ce5103 > > root@proxmox1:~# > > Our glusterfs storage backend consists of three servers all running Ubun= tu > 22.04 and glusterfs version 10.1. There are no errors in the logs on the > glusterfs hosts when a VM crashes and because some times also icinga plu= gins > get corupted I do get a very exact time range to search in the logs for > errors and warnings. > > However, I think it has something to do with our glusterfs setup. If I c= lone > a VM from a template I get the following: > > root@proxmox1:~# qm clone 9000 200 --full --name testvm --description > "testvm" --storage gfs_vms = = [62/62] > create full clone of drive ide2 (gfs_vms:9000/vm-9000-cloudinit.qcow2) > Formatting > 'gluster://gluster1.linova.de/gfs_vms/images/200/vm-200-cloudinit.qcow2'= , fmt=3Dqcow2 cluster_size=3D65536 extended_l2=3Doff preallocation=3Dmetad= ata compression_type=3Dzlib size=3D4194304 lazy_refcounts=3Doff refcount_b= its=3D16 > [2023-05-30 16:18:17.753152 +0000] I > [io-stats.c:3706:ios_sample_buf_size_configure] 0-gfs_vms: Configure ios= _sample_buf size is 1024 because ios_sample_interval is 0 > [2023-05-30 16:18:17.876879 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-0:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:17.877606 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-1:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:17.878275 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-2:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:27.761247 +0000] I [io-stats.c:4038:fini] 0-gfs_vms: > io-stats translator unloaded > [2023-05-30 16:18:28.766999 +0000] I > [io-stats.c:3706:ios_sample_buf_size_configure] 0-gfs_vms: Configure ios= _sample_buf size is 1024 because ios_sample_interval is 0 > [2023-05-30 16:18:28.936449 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-0: > All subvolumes are down. Going offline until at least one of them comes = back up. > [2023-05-30 16:18:28.937547 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-1:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:28.938115 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-2:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:38.774387 +0000] I [io-stats.c:4038:fini] 0-gfs_vms: > io-stats translator unloaded > create full clone of drive scsi0 (gfs_vms:9000/base-9000-disk-0.qcow2) > Formatting > 'gluster://gluster1.linova.de/gfs_vms/images/200/vm-200-disk-0.qcow2', f= mt=3Dqcow2 cluster_size=3D65536 extended_l2=3Doff preallocation=3Dmetadata= compression_type=3Dzlib size=3D10951327744 lazy_refcounts=3Doff refcount_= bits=3D16 > [2023-05-30 16:18:39.962238 +0000] I > [io-stats.c:3706:ios_sample_buf_size_configure] 0-gfs_vms: Configure ios= _sample_buf size is 1024 because ios_sample_interval is 0 > [2023-05-30 16:18:40.084300 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-0:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:40.084996 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-1:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:40.085505 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-2:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:49.970199 +0000] I [io-stats.c:4038:fini] 0-gfs_vms: > io-stats translator unloaded > [2023-05-30 16:18:50.975729 +0000] I > [io-stats.c:3706:ios_sample_buf_size_configure] 0-gfs_vms: Configure ios= _sample_buf size is 1024 because ios_sample_interval is 0 > [2023-05-30 16:18:51.768619 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-0:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:51.769330 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-1:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:18:51.769822 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-2:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:19:00.984578 +0000] I [io-stats.c:4038:fini] 0-gfs_vms: > io-stats translator unloaded > transferred 0.0 B of 10.2 GiB (0.00%) > [2023-05-30 16:19:02.030902 +0000] I > [io-stats.c:3706:ios_sample_buf_size_configure] 0-gfs_vms: Configure ios= _sample_buf size is 1024 because ios_sample_interval is 0 > transferred 112.8 MiB of 10.2 GiB (1.08%) > transferred 230.8 MiB of 10.2 GiB (2.21%) > transferred 340.5 MiB of 10.2 GiB (3.26%) > ... > transferred 10.1 GiB of 10.2 GiB (99.15%) > transferred 10.2 GiB of 10.2 GiB (100.00%) > transferred 10.2 GiB of 10.2 GiB (100.00%) > [2023-05-30 16:19:29.804006 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-0:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:19:29.804807 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-1:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:19:29.805486 +0000] E [MSGID: 108006] > [afr-common.c:6140:__afr_handle_child_down_event] 0-gfs_vms-replicate-2:= All subvolumes are down. Going offline until at least one of them comes b= ack up. > [2023-05-30 16:19:32.044693 +0000] I [io-stats.c:4038:fini] 0-gfs_vms: > io-stats translator unloaded > root@proxmox1:~# > > Is this message about the subvolumes which are down normal or might this= be > the reason for our strange problems? > > I have no idea how to further debug the problem so any helping idea or h= int > would be great. Pleae let me also know if I can provide more infos regar= ding > our setup. > > Ciao and thanks a lot, > > Schoepp >