From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 387321FF39B for ; Mon, 17 Jun 2024 17:58:04 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0677D74B7; Mon, 17 Jun 2024 17:58:06 +0200 (CEST) Message-ID: Date: Mon, 17 Jun 2024 17:58:00 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta To: Dominik Csapak , Proxmox Backup Server development discussion References: <20240611093046.1092265-1-d.csapak@proxmox.com> <6bbdf56a-a2ee-45d5-9645-2720f19757b2@proxmox.com> <30fce012-b3bd-4ba3-8912-08be43255004@proxmox.com> Content-Language: en-GB, de-AT From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: X-SPAM-LEVEL: Spam detection results: 0 AWL -0.048 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pbs-devel] [PATCH proxmox-backup] docs: add note for not using remote storages X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Am 13/06/2024 um 10:02 schrieb Dominik Csapak: > On 6/12/24 17:40, Thomas Lamprecht wrote: > but we already do recommend against using remote storage regularly, > just not in the docs but in the forum. (so do many of our users) > > we also recommend against slow storage, but that can also work > depending on the use case/workload/exact setup If a user complains it's safe to assume that it's too slow for their use case, otherwise they would not be in the forum. It's also OK to tell users that their storage is too slow and a local storage with some SSDs might be a (relatively) cheap alternative to address that, especially in the previous mentioned combination where a small and fast local storage is used for incoming backups while still using the remote storage to sync a longer history of backups too. Both have nothing to do with a blanket recommendation against remote storage, i.e., without looking at the actual setup closely, and I hope not that's such blanket statements are currently done frequently without context. >>> >>> (i know that datastore creation is not the best benchmark for this, >>> but shows that there is significant overhead on some operations) >> >> Yeah, one creates a datastore only once, and on actual backup there >> are at max a few mkdirs, not 65k, so not really relevant here. >> Also, just because there's some overhead, allowing simultaneous mounts >> doesn't come for free, it doesn't mean that it's actually a problem for >> actual backup. As said, a blanket recommendation against a setup that >> is already rather frequent is IMO just deterring (future) users. > > it's not only datastore creation, also garbage collection and > all operations that has to access many files in succession suffers > from the over head here. > > my point is that the overhead of using a remote fs (regardless which) > adds so much overhead that it often turns what would be 'reasonable' > performance locally into 'unreasonably slow' so you'd have to massively > overcompensate for that in hardware. This is possible ofc, but highly > unlikely for the vast majority of users. > That a storage being remote makes it unusable slow for PBS by definition is just not true (see next paragraph of my reply for expanding on that). >> >> If only; from forum and office request it's quite sensible to assume >> that a good amount of users already have their storage box, and they'd >> need to do so to be able to test it in any way, so already too late. >> >> It might be better to describe a setup how to still be able to use their >> existing, NFS/SMB/... attached storage in the best way possible. E.g., by >> doing a fast small local storage for incoming backups and use the bigger >> remote storage only through syncing to it. This has a few benefits beside >> getting good performance with existing, slower storage (of any type), like >> having already an extra copy of most recent data. > > ultimately it's your call, but personally i'd prefer a broad statement > that defers users from using a sub optimal setup in the first place > than not mentioning it at all in the official docs and explaining > every week in the forums that it's a bad idea Again, just because a storage is remote just does *not* mean that it has to be too slow to be used. I.e., just because there is _some_ overhead it does *not* mean that it will make the storage unusable. Ceph, e.g., is a remote storage that can be made plenty of fast, as our own benchmark papers how, and some users in huge environments even have to use it for backups as nothing else can scale amount of data and performance. Or take Blockbridge, they're providing fast remote storage through NVMe over TCP. So by counterexample, including our *own* benchmarks, I think we really can establish as a fact that there can be remote storage setups that are fast, and I do not see any point in arguing that further. > > this is the same as recommending fast disk, as one can use slow disks > in some (small) setups successfully without problems, but it does not > scale properly so we recommend against it. for remote storage, It really isn't, recommending for fast local storage in a recommended system specs section is not the same > the vast majority of users won't probably invest in a super > high performance nas/san box so recommending against using those > is worth mentioning in the docs IMHO As mentioned in my last reply, with that logic we have thousands+ things to recommend against, lots of old/low-power/ HW, some USB HW (some other nice one can be totally fine again), ... this would blow up the section such over some time, that almost nobody would read it to completion, not really helping such annoying cases in the forum or other channels (that cannot be really fixed by just adding a bulletin point, IME they're even encouraged to further go in the wrong direction if argumentation isn't sound (and sometimes even then..)). > > it does not have to be in the system requirements though, we could > also put a longer explanation in e.g. the FAQ or datastore section. > i just put it in the system requirements because we call out > slow disks there too and i guessed this is one of the more > read sections. > I reworked the system requirements part to my previous proposal, that fit's the style of recommending for things, not against, and tells the user what's actually important, not some possible correlation to that. https://git.proxmox.com/?p=proxmox-backup.git;a=commitdiff;h=5c15fb97b4d507c2f60428b3dba376bdbfadf116 This is getting long again and so only as short draft that would need some more thoughts and expansion, but a IMO better help that recommending against such things would be to provide a CLI command that allows users to test some basic throughput and access times (e.g. with cold/flushed FS cache) and use these measurements to extrapolate on some GC/Verify examples that try to mirror some real-world smaller/medium/big setups. While naturally still not perfect it would tell the user much more to see that a work load with, e.g., 30 VMs (backup group), with each say ~100 GB of space usage, and 10 snapshots per backup group each, would need roughly X time for a GC and Y time for a verification of all data. Surely quite a bit more complex to do sanely, but something like that would IMO *much* more helpful. _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel