From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4DE829868C for ; Sun, 12 Nov 2023 20:47:42 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 27117AF7B for ; Sun, 12 Nov 2023 20:47:12 +0100 (CET) Received: from gmmr-4.centrum.cz (gmmr-4.centrum.cz [IPv6:2a00:da80:1:502::8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Sun, 12 Nov 2023 20:47:10 +0100 (CET) Received: from gmmr-4.centrum.cz (localhost [127.0.0.1]) by gmmr-4.centrum.cz (Postfix) with ESMTP id D4C1C2A3EB for ; Sun, 12 Nov 2023 20:47:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=volny.cz; s=mail; t=1699818423; bh=zZkb5u1y8WwmQJ9BXbe2gUdTaaUJy25TwOmcSlYKYrA=; h=From:Subject:Date:References:To:In-Reply-To:From; b=lqNh2xj8o8r3UG2y4aSneMG+6FdZpXeFiYo4p/VJbrcDIYZ65VqmS952KdPV6nYMN 2WQH1szjIlQhRv+JUMjkRNF0cR6o5X2s30/q2YL5qdPgfNvv0Q3fF3KvfbK0hIMY7Z pL4T0XdKpTKOA1QHGZOwFMv2pMtHN4t3EW+DlOLs= Received: from antispam102.centrum.cz (unknown [10.30.208.102]) by gmmr-4.centrum.cz (Postfix) with ESMTP id D3649201A021 for ; Sun, 12 Nov 2023 20:47:03 +0100 (CET) X-IPAS-Result: =?us-ascii?q?A2EEAACAKlFl/03h/y5aGgEBAQEBAQEBAQEDAQEBARIBA?= =?us-ascii?q?QEBAgIBAQEBSYEyBQEBAQELAYMygV2EU4gdiUIDhDuNX4UUhlCBJQMYNQkPA?= =?us-ascii?q?QEBAQEBAQEBCS4BCgsEAQEDBIR/AocoJzQJDgECBAEBAQEDAgMBAQEBAQEDA?= =?us-ascii?q?QEGAQEBAQEBBgUBAoEZhS85DYJcgR1fCTYBAQEBAQEBAQEBAQEBARsCDXsBA?= =?us-ascii?q?QEBAgEBASFLEAsLDAELKgICJwEvBhODAAGCOyMUqmV6gTIaAmWzCR+BYgaBS?= =?us-ascii?q?AGIDAGBTgGIOQJCgg2BPAsQgjcxPoEFd2UBAQOBFy6DczmCLwSDbYU4BzKCI?= =?us-ascii?q?4MpKY1KXSJHWhYdAwcDVikQKwcEMBsHBgkUGBUjBlEEFxEkCRMSPgSBZIFRC?= =?us-ascii?q?n8/Dw4Rgj8iAgc2NhlIgSKBORUGOkp2ECoEFBdrIAgEahsVHjcREhcNAwh0H?= =?us-ascii?q?QIRIzwDBQMEMwoSDQshBRMBQgNCBkkLAwIaBQMDBIE2BQ0eAhAaBg0nAwMTT?= =?us-ascii?q?QIQFAM7AwMGAwsxAzBVRAxQA2sfGhwJPA8MHwIbHg0nKAI1QwMRBRICFgMkG?= =?us-ascii?q?QRFAwkDBwUsHUADCxgNSBEsNRQbBj8oSwedZg+CaHUHATAfAw4aEwEbEFsqC?= =?us-ascii?q?04tPTKSOo8/g06KLJRqhBeBX4JnhzuNG4gHBC+EAZNLAxiSHJg/jWWWEQGEW?= =?us-ascii?q?oFjaIEuTTg7KgGCPD8TGY47CwuDVoJkjRZ1OwIHAQoBAQMJiSKBQAEB?= IronPort-PHdr: A9a23:4uvx7RzzstRKuR/XCzKUzVBlVkEcU1XcAAcZ59Idhq5Udez7ptK+Z hyZvK89xwaTDc3y0LFttan/i+PaZSQ4+5GPsXQPItRndiQuroEopTEmG9OPEkbhLfTnPGQQF cVGU0J5rTngaRAGUMnxaEfPrXKs8DUcBgvwNRZvJuTyB4Xek9m72/q99pHNYAhEniexba1xI Rm5sAndqMobipZ+J6gszRfEvnRHd+NKyG1yIl6dgwjy7dqq8p559CRQtfMh98peXqj/Yq81U 79WAik4Pm4s/MHkugXNQgWJ5nsHT2UZiQFIDBTf7BH7RZj+rC33vfdg1SaAPM32Sbc0WSm+7 6puVRTlhjsLOyI//WrKkcF7kr5Vrwy9qBx+247UYZ+aNPxifqPGYNgWQXNNUttNWyBdB4+xa YwAAfccPeZDt4nyuUcBrQewCwmrAePg0D5Ihnnr1qE+3OksFQTK0Qo9FNwMrXvUts34OqcOX u+60afGwzvDYe1U1jn+74bFaQoure2JXb9qccrd1UsiGgXYhVuQs4zlODaV2/wJvWeB6eprS O2gi2o9pA5tojik28AhiovTiYIT11DE7j50wJgyJd28Tk56YcSoH4dRtiyDMYZ9X8wtTX1yt ikg1r0GpYC0fDIMyJk/xRPSdeKLfYeV7x79WuidPyl1iX1rdr+9mxu//0euxvHhWsSqzVtGs iVLn9vRu34D0xHe7tWKR+Z880mu2zuB2Qbe4fxKL0AzkKrUMZ8hwrgom5oWr0vDGzP2l1vtj K+KdUgp+/Kk5PjgYrXjvpOcOI50ih3+MqQzgcOzG+U4Mg8WU2iD4+S807Li/Uj+QLRFlPE5j q7ZsJXcKM8GpaC5GxdY34ki5hqlETur0NQVkWMZIF5beB+LlYrkNlPILfvlE/q/h0mskCx3x //YJL3vH5TNL3ndn7r5Ybpw7VNXxhApwtBF/Z1UD6kMIPf0WkDsqtPVFgQ5Mwmow+b/D9V9y 58eVXqPAqCHNKPStkeF5u0qI+WVf48VvSr9K+A46/Lyl3M2hUESfa+13ZsNdXC4G+5qLFmYY Hb0g9cBF3kFvhYmQeHliVCOSyBfanayUq4m+D03FoKrAZ3BS42vmLCB2T20HpxSZmBIEFCMF nLoep2GW/gWbiKSP9RhnycaVbimT48h0xGvuBXmxLtmNubU/DYXuor51Ndp/+3TiQ0y9TtsA siAy22NVX17nnsURz8q26ByuVR9ykmd3qRhnvxYDsJc5+lSXwc+NJ7c1PB6C8voVgLZZNeJT 1mnQtChATA+VN4+39gOY0MuU+ml2yjZ2DqvBK9dqZHDGYc56OrY3mPtb5Jl1XuDybIslXEiQ 9BTLiu+nKlv8g/DQYXOlhPKubytcPE+1SiFymOKxmrG6ENRVhNrebvCWXYOIEDE+4eqrnjeR qOjXOx0ejBKztSPf+4XM7XU IronPort-Data: A9a23:9D01P6D7NyFtIhVW/4rhw5YqxClBgxIJ4kV8jS/XYbTApDh2hTVWn GtOXT2EOKyMZmX2eoojOtnk9U8CvpCHz4NlOVdlrnsFo1CmCCbm6XZ1Cm+qYkt+++WaFBoPA /02M4WGdoZtJpPljk/FGqD7qnVh3r2/SLP5CerVUgh8XgYMpB0J0HqPoMZnxNYz6TSFK1nV4 4ir+pSFYAXNNwNcawr41YrS9XuDg9yt4Fv0jnRmDdhXsVnXkWUiDZ53Dcld+FOlKmX8NrfSq 9frlNlVzEuAl/seIo/NfoLAT6E/auW60T6m0SMKBvf40nCukQRpukozHKJ0hU66EFxllfgpo DlGncTYpQvEosQgMQnSOvVVO3gWAEFIxFPICXytkOWc1EDaSlC2365vF1AWIqIz2d8iVAmi9 dRAQNwMRh+Gwvmz3KriEq9ni8I/NtTueogN0p1i5W2HS6x7HNaZGfiMuoAwMDQY36iiGd7cb swEchJxaxDGcltEKD/7DbpixLbz3CilKVW0rnqWrIcdw1buwjAok76yYdz7U+G6TuJKyxPwS mXuuj6R7gshHMSBxH+Y6X+2ruvOhj/gHpoIE6298e4si1qWrlH/EzVKCx3h/KT/0BTjHY0AQ 6AJxhcTQWEJ3BTDZrHAs9eQ/BZoYjZ0twJsLtAH IronPort-HdrOrdr: A9a23:/GBaj6szqduQENkSCTGS3HzM7skDZ9V00zEX/kB9WHVpm62j+v xG88506faZslwssTQb6LO90de7MBHhHO9OjbX5VI3KNGLbUSmTTb2KhrGM/9SPIU3DH5ZmpM JdT5Q= X-Talos-CUID: =?us-ascii?q?9a23=3A02h9KWoUyUFsxOaQ8GxGgHfmUcQ3U1TS13rMGmT?= =?us-ascii?q?iFT1oU6y1VnC725oxxg=3D=3D?= X-Talos-MUID: =?us-ascii?q?9a23=3AsxikzQ6BsEkwtg9UZa+Mucsbxoxr6omEKH4wn6l?= =?us-ascii?q?f+M/bZCpeEgqthxOoF9o=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.03,298,1694728800"; d="scan'208,217";a="40932922" Received: from unknown (HELO gm-smtp10.centrum.cz) ([46.255.225.77]) by antispam102.centrum.cz with ESMTP; 12 Nov 2023 20:47:03 +0100 Received: from smtpclient.apple (unknown [10.128.64.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gm-smtp10.centrum.cz (Postfix) with ESMTPSA id D83D9B1F7E for ; Sun, 12 Nov 2023 20:47:02 +0100 (CET) From: Jan Vlach Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Date: Sun, 12 Nov 2023 20:47:01 +0100 References: <2E12D3CC-1D34-43F7-94B8-078C08078E8C@qwertz1.com> To: Proxmox VE user list In-Reply-To: <2E12D3CC-1D34-43F7-94B8-078C08078E8C@qwertz1.com> Message-Id: <5B3D932D-1AC8-4842-A14E-9B24D23FE67E@volny.cz> X-Mailer: Apple Mail (2.3696.120.41.1.4) X-SPAM-LEVEL: Spam detection results: 0 AWL 0.117 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] ZFS-8000-8A on a non-system disk. How to do? X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Nov 2023 19:47:42 -0000 Hi, having same number of checksum errors on all drives really means bad = cabling or bad RAM.=20 - if you have ECC RAM, check for errors in ipmitool i.e. ipmitool sel elist - you could see something in dmesg too. - if you don't have ECC ram, get the memtest in UEFI mode from = https://www.memtest86.com/ take the host = offline and let it run for day or two. - I've seen this with Supermicro server where the cable for last two = slots out of 10 was bent and touching the case lid and those two slots = have been resetting the bus showing me increasing errors on all drives. = Scrubs just changed the affected files and metadata, so I didn't trust = the host anymore and consistency of data, restored everything from good = backup to different one and then debugged. - If at this point you want to backup and restore and you don't have = backups, it's game over for you.=20 JV > On 12. 11. 2023, at 19:32, Stefan wrote: >=20 > I assume you already have ruled out flaky hardware? (Bad cable, RAM). = If so repairing is not possible. You can theoretically bypass the = backup/destroy/restore way but why? > You have three faulty drives that need to be replaced anyway. That = operation + identifying the failed file(s) takes much longer than just = copy back from backup. >=20 >=20 >=20 > Am 12. November 2023 16:21:17 MEZ schrieb Marco Gaiarin = : >>=20 >> I've got: >>=20 >> root@lisei:~# zpool status -xv >> pool: rpool-hdd >> state: ONLINE >> status: One or more devices has experienced an error resulting in = data >> corruption. Applications may be affected. >> action: Restore the file in question if possible. Otherwise restore = the >> entire pool from backup. >> see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A >> scan: scrub repaired 0B in 00:37:50 with 1 errors on Sun Nov 12 = 01:01:53 2023 >> config: >>=20 >> NAME STATE READ = WRITE CKSUM >> rpool-hdd ONLINE 0 = 0 0 >> raidz1-0 ONLINE 0 = 0 0 >> ata-WDC_WD2003FZEX-00SRLA0_WD-WMC6N0D6J2LN ONLINE 0 = 0 2 >> ata-WDC_WD2003FZEX-00SRLA0_WD-WMC6N0D7Z60F ONLINE 0 = 0 2 >> ata-WDC_WD2003FZEX-00SRLA0_WD-WMC6N0D2JSHZ ONLINE 0 = 0 2 >>=20 >> errors: Permanent errors have been detected in the following files: >>=20 >> rpool-hdd/vm-401-disk-0:<0x1> >>=20 >> disk is for an VM used as a mere repository for rsnapshot backup, so = contain >> many copy of the same files, with different and abunndant retention. >> Is an addon disk for the VM, eg i can safely if needed umount it, = even >> detach it. >>=20 >>=20 >> There's something i can do to repair the volue, possibly online? = really i >> have to backup it, destroy and restore from backup?! >>=20 >>=20 >> Thanks. >>=20 >> --=20 >> Non pu=C3=B2 sentirsi degno di essere italiano chi non vota SI al = referendum >> (Silvio Berlusconi, 21 giugno 2006) >>=20 >>=20 >>=20 >> _______________________________________________ >> pve-user mailing list >> pve-user@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>=20 > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user