From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DA59160503 for ; Mon, 10 Jan 2022 15:32:29 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D018A2F8C for ; Mon, 10 Jan 2022 15:32:29 +0100 (CET) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0731.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::731]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 6F92F2F20 for ; Mon, 10 Jan 2022 15:32:28 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Pr/NG8i1pTyGJpNCAadNckzMduY3pR6Z+XHGyUM/ZwKNN1nbAlY5Ri2XBgQh8OokmUkemv1YMzi9b4Tpn6Xevh3QkOMk/darO7mHuef1Ha7TScwbOcPb8l0SFWvpdLVlrWf/rUB7Uwuf6XHWNkLIBU+ELcY9GOsqAXQGw4Zzd8jYxGxp6x/1RBdfv3n2PHxCcAODHBWp3inpwmf2lG2GPOjx1SAdfpfvz4FR6Ua8Y4DP2iOj4sLTnlIxZy2G0adjHqXqk9C7FjGpWC4aDWpS9W8urla1QCMSOVEhqF6kv7CFIh7yuU2VhZo5LM9NOIJpXgkcRrGS2ypL+nX45dUUjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=g/X47RMIz/6o5nPz4ViXCBCj3lZGwt2NTQi7kyTQC+E=; b=SGyCKA637nfuGd2lI1+75nAdwWSLan0hEafNMQ4moEiHEToG+YktZ1YLoxj9+HO16pFD+ojNJvU4o6ZMbODx1f3mkOR5eueKXplr17CKzSRavKZ+S5KqkK5IEHm2K8cPyw7wEuclfOHlqMNAdOkdAGbZPcZq4YHvRYCcaU/Be5MQ4NlhSgTkC2wrNaHfwnEvEWEV14RIeK3C0GJ+O0nyaDb6w6VeUdE8egi+BtI21OgsE6/ugmX3pqFPbXZh5yKD/Uno+0mqVGnuw2/dSCIT2eKa9ouadv+cTqymeZ4lgO1EWzgt2uJpAWCSvmdi2bObCRx8DZmFHxbHEDGJs917SQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=athompso.net; dmarc=pass action=none header.from=athompso.net; dkim=pass header.d=athompso.net; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=BGETCA.onmicrosoft.com; s=selector2-BGETCA-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g/X47RMIz/6o5nPz4ViXCBCj3lZGwt2NTQi7kyTQC+E=; b=ahleyT8wP734pMxYwbc0aBiqLTSU0qSFSuDMpUPkGefJr3H58se6Ri5kfFhyF0zFKmH+0i8j5wuzV9fZ+iGUunMWjsXXjrUmA9HnO9MmE4O49LdqKkRM8LR18w1gRq667iSTF+n3nUpTqXOOkbPQmp+/16gveQi2FzFv+EhOorE= Received: from YT2PR01MB4622.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:3d::9) by YTOPR0101MB1899.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:23::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4867.11; Mon, 10 Jan 2022 14:32:20 +0000 Received: from YT2PR01MB4622.CANPRD01.PROD.OUTLOOK.COM ([fe80::58bf:8950:d8f0:694]) by YT2PR01MB4622.CANPRD01.PROD.OUTLOOK.COM ([fe80::58bf:8950:d8f0:694%2]) with mapi id 15.20.4867.012; Mon, 10 Jan 2022 14:32:19 +0000 From: Adam Thompson To: Proxmox VE user list , "pve-user@pve.proxmox.com" Thread-Topic: [PVE-User] Proxmox and ZFS on large JBOD ? Thread-Index: AQHYBiT4dFO4XOQ9XkCNMRBFtnEt5axcT/gy Date: Mon, 10 Jan 2022 14:32:19 +0000 Message-ID: References: <20220110141350.716d9727@schleppmd.hlrs.de> In-Reply-To: <20220110141350.716d9727@schleppmd.hlrs.de> Accept-Language: en-CA, en-US Content-Language: en-CA X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=athompso.net; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 23ebb903-9e87-438d-9ebb-08d9d4460235 x-ms-traffictypediagnostic: YTOPR0101MB1899:EE_ x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2201; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 25yUvLtvjLknPGY92uerC0x1d8s2yJWRCp4TC+uHpJNbF4UQLYGe2D7V1No+2h3c9GZm8EXFZR9UfNcX9EECGo1QeuAq3e9XdVY3g8RNj05/9ecWmwzT+m6rOqKpDM8SVl99MyjGzL1XCOZA2OLMpFKCdUT+t1IYWhSZmBg51oEQKTmaief6lBjNIxHVma6agXdD+Fo6uhID3IOthqr92FpkJV4JmAhXJr1J9hMOZ3WIN0PErr2QEBnH/5r0PIV+dBU4MFrVPOLmNah4ejUxegawNo8xD4t7Nq6cU2fC6cxd7JKyDUuGkJhIMBCqDipyrl3ftZ1f2co16Uak3Dc1NXZMCoDQCw0+HDKLIW0zFELEcUn6UM5vhGBx5ydXkYVltZRq2ROh7qch3jZvBLGDp8wbGksN1Pn+zpr+m8L/snMEU5B7E4i8/jsiMgT2+jLUf9W+uYnABfAqd4EVGZXn/RYoHmXWmXPU6vNzdFRePYpaOcwPFWaFpTzcmd4+sfhD0DbuUe0Jooukd2ALAvghjXbqqjFb2H3WZzWUa7ZnhUMe4S2k1fWdOifizwwHNv9+kd9aQP8yBcpy/HhCaZHHSGIVhrybNBSX9bGJnS2HtsVr8jc2ezFnyGOegdisp2PzGXhv/LM38N/tnAaajSahnOYaeM0owytjwIPATXs0LWbmCPYd1cKJyuzTWLGK3nnIbVu+WGdQ2es9UOLE9fBBHfMf6M1iPeBnhfkklhaBuzAx5GUWgad/cdvLhZ4a839tqnm2IN0S4prv640wXvwrhZwxfiRHSMgpgqBEvPVdrEc= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YT2PR01MB4622.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(376002)(346002)(39830400003)(136003)(366004)(396003)(76116006)(66556008)(64756008)(316002)(66446008)(71200400001)(166002)(7696005)(26005)(186003)(2906002)(110136005)(53546011)(966005)(33656002)(45080400002)(52536014)(122000001)(38100700002)(9686003)(5660300002)(6506007)(66946007)(38070700005)(8936002)(86362001)(508600001)(55016003)(8676002)(66476007); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?RCSkfyBKeNq29NFy8joJH4DqbYndH7OmJjCGDvB3wkzdpjOQA0UWYjWQY1yt?= =?us-ascii?Q?FdVMZlFYTPOnJtrMvh4CLVAXvnh214YkMulRF5CdEiHJPFJoCQs/qk5RG3tw?= =?us-ascii?Q?AYUk2dFlI5KTgJ04cN7Bh76Uj3uPLbKFqw0XOsgqCVRoz+1lvqczumla2err?= =?us-ascii?Q?4exwuuw5NDfWiNT2kvqUeBZFRvak9+RSHKrUcbjflwxOA5/Dr9oO9em8WmH+?= =?us-ascii?Q?P7CijcFquAkg9MuKNVFK78YMV+PM/h2I5W6S/icbhw70pnXRmqG10cHXGbOS?= =?us-ascii?Q?azgWg7oHMjd1qCjCFWqYcIexRbzap2N1BKEbVS6BhbbR7usx++O3n6Am5bbk?= =?us-ascii?Q?V6xE2hggA7JdwNVEDrLWbAjiVhGRyV7ZKKfLq2r+fUtMf+dVj4Owi4iB2Nmz?= =?us-ascii?Q?9tTlfHxMujKu5gspWj85c1z0GKbClc8D0u6tKluBbkoufR7Jtlc8KMp1q21u?= =?us-ascii?Q?EEKmrnsT6NQ2nw0eLcstJPnBrfvH0xVgzLSE+wVb1JoKLRWfZMuka1rQINj+?= =?us-ascii?Q?PXCMi5GEy0mpF+sc8l6DryKr6yxP/zejaR7CN+60w34Vm96X2m0DL2MTddrq?= =?us-ascii?Q?/qBErXjcTz2PQvaROYLtoajdnKOAjgaibiwGUq3uYHHoot7VHbIhmTlpnuLw?= =?us-ascii?Q?fRG032pSADwcrMuTM0BVsnp952J+ckeIDx7nVRIjABtdsR9kH6U8A8TwxvWF?= =?us-ascii?Q?Vgqkc7t3i7ZQsHJsTcQFw+1uam790MtP1mkFcMoIA4DEwFVfd9wWUfB58N5E?= =?us-ascii?Q?Xdb8SVTfQQADPyoN6iJdHRgAY0Je3QYHR/VBmZvz1dxZgYsi/dQ/phCImlsK?= =?us-ascii?Q?mP+/gP9qMumKoJZZp6paz6ydmPkYLYQ2bNJjwHxyUxaKqHK6NNuMsXmMqLEG?= =?us-ascii?Q?w8zJUg2Jb67pX4ecblLkaQhHDNMh5kVtjfX85UXYHkxegSJAobkz7ux1QlvN?= =?us-ascii?Q?E/oAs7xOiivxxzgB2aehx+jNTapk4YVY+suMiAdM9ScHF+EA7n0F2Hj3twSh?= =?us-ascii?Q?9MNG5rux21e+p3RiOJocSgLPCAo4KTbO55eVlLDbL8A7T8ktBt8ntK4czw5c?= =?us-ascii?Q?PzJoUJKiANz1IdqAgxNEtelH/u3xB/WK1DGTsJBE6b5c87kWYKHh4lDesXcw?= =?us-ascii?Q?hfMutqLoUdupUi4YKDIBS4DWEIXdp2JRC4zHl7jSAcmfm90eGh2zd7DNlCXX?= =?us-ascii?Q?cSsI6PeH0ddTLDa+OrWo2EV/PpyThOz5ZrngV0YINfWvZ9a2evZYA0/hx8uy?= =?us-ascii?Q?H+fDARL5NKylZoLtkU1DO2l3DQyzg5vwJFwhHXsyAuqMemeDV38utSDuVK1q?= =?us-ascii?Q?M0DqSo7ngvONMbzbiOMbcGWPqD6GtSbsJHSSSRmEI7uPuOdzG4qnCqow1dvf?= =?us-ascii?Q?8xJaKyxU0VpAyKXPxRLzHwSAL0UvpMGHl+HyzLwBQloLMTcJ6ghnXiO+WWUi?= =?us-ascii?Q?zfqJ5GSR5SYp/0iQdw+Kclx7GDV7+S7ow9IsQwEfyQCqBGhqZE7/uR3QV+5h?= =?us-ascii?Q?apBrGsnD0OO+XuOj8wprhApy2pdVVwrFAVgJgFM4B4syWEhYYYt1laayye5G?= =?us-ascii?Q?pURcD+BJs/dJ9qtdRLWr9Eq0NpkvBH5YxmulLJVIFIx4bj6/pqei0zgs99uT?= =?us-ascii?Q?T3rZoYpRtuOCFd19KNlffTA40IlaU3VXUxHYxUmxTG6L?= MIME-Version: 1.0 X-OriginatorOrg: athompso.net X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB4622.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 23ebb903-9e87-438d-9ebb-08d9d4460235 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jan 2022 14:32:19.8566 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 10d7fd83-719f-4fa4-ab6b-855821d5b188 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: eJWqgtSu2zG5OhtVrPvKCJ50GlGTyrdhIZ4xM9lnCfSFJER9J5kCH7XZbOVpMez6oUWynZ5ZXd2ZyvoWC3EChw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1899 X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature HTML_MESSAGE 0.001 HTML included in message KAM_SHORT 0.001 Use of a URL Shortener for very short URL SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] Proxmox and ZFS on large JBOD ? X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2022 14:32:29 -0000 IMHO, you want TrueNAS, not Proxmox, to solve this problem. Ultimately, the problem is SystemD's notion of dependencies and timeouts, w= hich Proxmox+OpenZFS still relies on. Source: I have a Debian 10 system with 29 storage devices, 24 of which are = multipathed, and have had to edit & override various systemd settings to ge= t it to boot cleanly, reliably. -Adam Get Outlook for Android ________________________________ From: pve-user on behalf of Martin Dzi= obek Sent: Monday, January 10, 2022 7:13:50 AM To: pve-user@pve.proxmox.com Subject: [PVE-User] Proxmox and ZFS on large JBOD ? Hi pve-users ! Does anybody has experiences if proxmox works flawlessly to manage a large zfs volume consisting of a SAS-connected JBOD of 60 * 1TB-HDDs ? Right now, management is done with a regular Debian 11 installation, and rebooting the thing always ends up in a timeout mess at network startup, because it takes ages to enumerate all those member disks, import the zpool and export it via NFS. I am considering to install Proxmox on this server for the sole purpose of smooth startup and management operation. Might that be a stable solution ? Best regards, Martin _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user