From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pbs-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 173781FF16B for <inbox@lore.proxmox.com>; Thu, 6 Mar 2025 15:02:48 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E01CE7EA3; Thu, 6 Mar 2025 15:02:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=tuxis.nl; s=mail; h=from:reply-to:subject:date:message-id:to:cc:mime-version:content-type: in-reply-to:references; bh=P+of+ntNIcs9zLYbTUGMGXXBjWaR6b3/xxrPLNjLKWw=; b=OwnuHVvbl6Vqsym+D9tVrcqlytFXMkZZLEXfPHpPBDYJXQyjudQTJTAyNsONlMXXYs9PUOhw9QZnY oHfvdvZSUfbB6zMzoDYdzS7xJkx+O8qOFt3y9WmG/tFWr7cmltvNOgg8tqZlvh0Yww/lU6J3doGgnQ 24IGIQs4/vaXIZM+zfWnC04/zoFvvWq0zbi0a+rQejRVmoELvfnDep1zMexCTcsNmbsln3knhPx2Ea lm0cphEmkUvPjKIv0WIu4KvQXgS+fkqx5M1sQ2NMT/chkbTroigqJxQtvURj3DBc7zJ+bA58J9oR+4 a18kqdWh6fVqWlUTtPOa5gs3M5kftJQ== X-Footer: dHV4aXMubmw= From: "Mark Schouten" <mark@tuxis.nl> To: "Shannon Sterz" <s.sterz@proxmox.com> Date: Thu, 06 Mar 2025 13:32:15 +0000 Message-Id: <em372c8493-fc98-4f26-af91-658a1b05061e@fb5eb041.com> In-Reply-To: <em808af812-c25f-4569-8e1c-231663ec413f@d1b4cc0c.com> References: <ema6032dba-8585-4377-bec1-11a37159087c@192dede7.com> <D6D024F1JMI5.QDFXDKCQMUCJ@proxmox.com> <embeb48874-d400-4e69-ae0f-2cc56a39d592@93f95f61.com> <D6D3QC6Y5H4S.1QHYHPHXK6RVR@proxmox.com> <em2d4327d4-4607-458e-880d-9b6bfba58f0a@ef757d37.com> <D6GK5TIN2LD4.1AXEF95IHZITS@proxmox.com> <em808af812-c25f-4569-8e1c-231663ec413f@d1b4cc0c.com> User-Agent: eM_Client/10.1.4828.0 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.105 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy HTML_MESSAGE 0.001 HTML included in message KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] Authentication performance X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Mark Schouten <mark@tuxis.nl>, Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Cc: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Content-Type: multipart/mixed; boundary="===============6086177299285116599==" Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com> --===============6086177299285116599== Content-Type: multipart/alternative; boundary="------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7" --------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi, Is there anything I can do to make this bug go up on the priority list?=20 :) We=E2=80=99ve currently stopped supplying free accounts to the masses, beca= use=20 we are pretty sure that it will cause more and more issues to our setup=20 if we keep creating more datastores and stuff. Please let me know.. =E2=80=94 Mark Schouten CTO, Tuxis B.V. +31 318 200208 / mark@tuxis.nl ------ Original Message ------ >From "Mark Schouten" <mark@tuxis.nl> To "Shannon Sterz" <s.sterz@proxmox.com> Cc "Proxmox Backup Server development discussion"=20 <pbs-devel@lists.proxmox.com> Date 06/01/2025 20:07:43 Subject Re[6]: [pbs-devel] Authentication performance >https://bugzilla.proxmox.com/show_bug.cgi?id=3D6049 has been created for= =20 >this. > >Thanks! > >=E2=80=94 >Mark Schouten >CTO, Tuxis B.V. >+31 318 200208 / mark@tuxis.nl > > >------ Original Message ------ >>From "Shannon Sterz" <s.sterz@proxmox.com> >To "Mark Schouten" <mark@tuxis.nl> >Cc "Proxmox Backup Server development discussion"=20 ><pbs-devel@lists.proxmox.com> >Date 20/12/2024 14:22:18 >Subject Re: Re[4]: [pbs-devel] Authentication performance > >>On Thu Dec 19, 2024 at 10:56 AM CET, Mark Schouten wrote: >>> Hi, >>> >>> We upgraded to 3.3 yesterday, not much gain to notice with regards to >>> the new version or the change in keying. It=E2=80=99s still (obvioulsy= ) pretty >>> busy. >> >>just be aware that the patch i linked to in my last mail has not been >>packaged yet, so you wouldn't see the impact of that patch yet. >> >>> However, I also tried to remove some datastores, which failed with >>> timeouts. PBS even stopped authenticating (so probably just working) a= ll >>> together for about 10 seconds, which was an unpleasant surprise. >>> >>> So looking into that further, I noticed the following logging: >>> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >>> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >>> [::ffff]:42104] Unable to acquire lock >>> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err= or >>> 4) >>> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >>> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >>> [::ffff]:42144] Unable to acquire lock >>> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err= or >>> 4) >>> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >>> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >>> [::ffff]:47286] Unable to acquire lock >>> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err= or >>> 4) >>> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >>> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >>> [::ffff:]:45994] Unable to acquire lock >>> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err= or >>> 4) >>> >>> Which surprised me, since this is a =E2=80=99status=E2=80=99 call, whi= ch should not need >>> locking of the datastore-config. >>> >>>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/ad= min/datastore.rs;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l68= 7 >>> does not lock the config, but >>> >>>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-datasto= re/src/datastore.rs;h=3D0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=3DHEAD#= l204 >>> does. >>> >>> So if I understand this correctly, every =E2=80=99status=E2=80=99 call = (30 per second in >>> our case) locks the datastore-config exclusively. And also, every time >>> =E2=80=99status=E2=80=99 get called, the whole datastore-config gets l= oaded? >> >>probably, there are some comments about that there already, it might >>make sense to open a bugzilla issue to discuss this further [1]. >> >>[1]: https://bugzilla.proxmox.com/ >> >>> Is that something that could use some performance tuning? >>> >>> =E2=80=94 >>> Mark Schouten >>> CTO, Tuxis B.V. >>> +31 318 200208 / mark@tuxis.nl >>> >>> >>> ------ Original Message ------ >>> From "Shannon Sterz" <s.sterz@proxmox.com> >>> To "Mark Schouten" <mark@tuxis.nl> >>> Cc "Proxmox Backup Server development discussion" >>> <pbs-devel@lists.proxmox.com> >>> Date 16/12/2024 12:51:47 >>> Subject Re: Re[2]: [pbs-devel] Authentication performance >>> >>> >On Mon Dec 16, 2024 at 12:23 PM CET, Mark Schouten wrote: >>> >> Hi, >>> >> >>> >> > >>> >> >would you mind sharing either `authkey.pub` or the output of the >>> >> >following commands: >>> >> > >>> >> >head --lines=3D1 /etc/proxmox-backup/authkey.key >>> >> >cat /etc/proxmox-backup/authkey.key | wc -l >>> >> >>> >> -----BEGIN RSA PRIVATE KEY----- >>> >> 51 >>> >> >>> >> So that is indeed the legacy method. We are going to upgrade our P= BS=E2=80=99es >>> >> on wednesday. >>> >> >>> >> > >>> >> >The first should give the PEM header of the authkey whereas the s= econd >>> >> >provides the amount of lines that the key takes up in the file. B= oth >>> >> >give an indication whether you are using the legacy RSA keys or n= ewer >>> >> >Ed25519 keys. The later should provide more performance, security = should >>> >> >not be affected much by this change. If the output of the command= s look >>> >> >like this: >>> >> > >>> >> >-----BEGIN PRIVATE KEY----- >>> >> >3 >>> >> > >>> >> >Then you are using the newer keys. There currently isn't a recomm= ended >>> >> >way to upgrade the keys. However, in theory you should be able to = remove >>> >> >the old keys, re-start PBS and it should just generate keys in th= e new >>> >> >format. Note that this will logout anyone that is currently >>> >> >authenticated and they'll have to re-authenticate. >>> >> >>> >> Seems like a good moment to update those keys as well. >>> > >>> >Sure, just be aware that you have to manually delete the key before >>> >restarting the PBS. Upgrading alone won't affect the key. Ideally you= 'd >>> >test this before rolling it out, if you can >>> > >>> >> >In general, tokens should still be fater to authenticate so we'd >>> >> >recommend that you try to get your users to switch to token-based >>> >> >authentication where possible. Improving performance there is a b= it >>> >> >trickier though, as it often comes with a security trade-off (in= the >>> >> >background we use yescrypt fo the authentication there, that >>> >> >delibaretely adds a work factor). However, we may be able to impr= ove >>> >> >performance a bit via caching methods or similar. >>> >> >>> >> Yes, that might help. I=E2=80=99m also not sure if it actually is >>> >> authentication, or if it is the datastore-call that the PVE-enviro= nments >>> >> call. As you can see in your support issue 3153557, it looks like= some >>> >> requests loop through all datastores, before responding with a lim= ited >>> >> set of datastores. >>> > >>> >I looked at that ticket and yes, that is probably unrelated to >>> >authentication. >>> > >>> >> For instance (and I=E2=80=99m a complete noob wrt Rust) but if I u= nderstand >>> >>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api= 2/admin/datastore.rs;h=3D11d2641b9ca2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD= #l1315 >>> >> correcly, PBS loops through all the datastores, checks mount-statu= s and >>> >> config, and only starts filtering at line 1347. If I understand th= at >>> >> correctly, in our case with over 1100 datastores, that might cause = quite >>> >> some load? >>> > >>> >Possible, yes, that would depend on your configuration. Are all of th= ese >>> >datastores defined with a backing device? Because if not, than this >>> >should be fairly fast (as in, this should not actually touch the disk= s). >>> >If they are, then yes this could be slow as each store would trigger= at >>> >least 2 stat calls afaict. >>> > >>> >In any case, it should be fine to move the `mount_status` check after >>> >the `if allowed || allow_id` check from what i can tell. Not sure why >>> >we'd need to check the mount_status for a datastore we won't include= in >>> >the resulsts anyway. Same goes for parsing the store config imo. Send = a >>> >patch for that [1]. >>> > >>> >[1]: https://lore.proxmox.com/pbs-devel/20241216115044.208595-1-s.ste= rz@proxmox.com/T/#u >>> > >>> >> >>> >> >>> >> Thanks, >>> >> >>> >> =E2=80=94 >>> >> Mark Schouten >>> >> CTO, Tuxis B.V. >>> >> +31 318 200208 / mark@tuxis.nl >>> > >>> > >> >> --------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head> <style type=3D"text/css"><!--#x3ccf03dbbf4f4dc blockquote.cite {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p= x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb= (204, 204, 204);} #x3ccf03dbbf4f4dc blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p= x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb= (204, 204, 204); margin-top: 3px; padding-top: 0px;} #x3ccf03dbbf4f4dc {font-family: Helvetica; font-size: 9pt;} --></style><style id=3D"css_styles" type=3D"text/css"><!--blockquote.cite { = margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px= ; border-left: 1px solid #cccccc } blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px;= padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding= -top: 0px; } a img { border: 0px; } table { border-collapse: collapse; } li[style=3D'text-align: center;'], li[style=3D'text-align: center; '], li[s= tyle=3D'text-align: right;'], li[style=3D'text-align: right; '] { list-sty= le-position: inside;} body { font-family: Helvetica; font-size: 9pt; } .quote { margin-left: 1em; margin-right: 1em; border-left: 5px #ebebeb soli= d; padding-left: 0.3em; } a.em-mention[href] { text-decoration: none; color: inherit; border-radius:= 3px; padding-left: 2px; padding-right: 2px; background-color: #e2e2e2; } --></style></head> <body style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; line-br= eak: after-white-space;"><div>Hi,</div><div><br /></div><div>Is there anyth= ing I can do to make this bug go up on the priority list?=C2=A0<span style= =3D"color: rgb(210, 210, 210); font-size: 9pt;">:)</span></div><div><span><= br /></span></div><div>We=E2=80=99ve currently stopped supplying free accou= nts to the masses, because we are pretty sure that it will cause more and m= ore issues to our setup if we keep creating more datastores and stuff.</div= ><div><br /></div><div>Please let me know..</div><div><br /></div><div id= =3D"signature_old" style=3D"clear:both"><div style=3D"margin: 0px; padding: = 0px; box-sizing: content-box;">=E2=80=94=C2=A0</div><div style=3D"margin:= 0px; padding: 0px; box-sizing: content-box;">Mark Schouten</div><div style= =3D"margin: 0px; padding: 0px; box-sizing: content-box;">CTO, Tuxis B.V.</d= iv><div style=3D"margin: 0px; padding: 0px; box-sizing: content-box;">+31 3= 18 200208 /=C2=A0mark@tuxis.nl</div></div><div><br /></div> <div x-em-replyforwardheader=3D""><br /></div> <div> <div>------ Original Message ------</div> <div>From "Mark Schouten" <<a href=3D"mailto:mark@tuxis.nl">mark@tuxis.n= l</a>></div> <div>To "Shannon Sterz" <<a href=3D"mailto:s.sterz@proxmox.com">s.sterz@= proxmox.com</a>></div> <div>Cc "Proxmox Backup Server development discussion" <<a href=3D"mailt= o:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>></div> <div>Date 06/01/2025 20:07:43</div> <div>Subject Re[6]: [pbs-devel] Authentication performance</div></div><div= x-em-quote=3D""><br /></div> <div id=3D"x3ccf03dbbf4f4dc" style=3D"overflow-wrap: break-word; -webkit-nb= sp-mode: space; line-break: after-white-space;"><blockquote cite=3D"em808af= 812-c25f-4569-8e1c-231663ec413f@d1b4cc0c.com" type=3D"cite" class=3D"cite2"= > <div><a href=3D"https://bugzilla.proxmox.com/show_bug.cgi?id=3D6049">https:= //bugzilla.proxmox.com/show_bug.cgi?id=3D6049</a>=C2=A0has been created for = this.</div><div><br /></div><div>Thanks!</div> <div><br /></div><div id=3D"signature_old" style=3D"clear:both"><div style= =3D"margin: 0px; padding: 0px; box-sizing: content-box;">=E2=80=94=C2=A0</d= iv><div style=3D"margin: 0px; padding: 0px; box-sizing: content-box;">Mark= Schouten</div><div style=3D"margin: 0px; padding: 0px; box-sizing: content-= box;">CTO, Tuxis B.V.</div><div style=3D"margin: 0px; padding: 0px; box-siz= ing: content-box;">+31 318 200208 /=C2=A0mark@tuxis.nl</div></div><div><br= /></div> <div x-em-replyforwardheader=3D""><br /></div> <div> <div>------ Original Message ------</div> <div>From "Shannon Sterz" <<a href=3D"mailto:s.sterz@proxmox.com">s.ster= z@proxmox.com</a>></div> <div>To "Mark Schouten" <<a href=3D"mailto:mark@tuxis.nl">mark@tuxis.nl<= /a>></div> <div>Cc "Proxmox Backup Server development discussion" <<a href=3D"mailt= o:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>></div> <div>Date 20/12/2024 14:22:18</div> <div>Subject Re: Re[4]: [pbs-devel] Authentication performance</div></div><= div x-em-quote=3D""><br /></div> <div id=3D"xd7a8ca7f59be40f" class=3D"plain"><blockquote cite=3D"D6GK5TIN2L= D4.1AXEF95IHZITS@proxmox.com" type=3D"cite" class=3D"cite2"> <div class=3D"plain_line">On Thu Dec 19, 2024 at 10:56 AM CET, Mark Schoute= n wrote:</div> <blockquote type=3D"cite" class=3D"cite"> <div class=3D"plain_line"> Hi,</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> We upgraded to 3.3 yesterday, not much gain to n= otice with regards to</div> <div class=3D"plain_line"> the new version or the change in keying. It=E2= =80=99s still (obvioulsy) pretty</div> <div class=3D"plain_line"> busy.</div> </blockquote> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">just be aware that the patch i linked to in my la= st mail has not been</div> <div class=3D"plain_line">packaged yet, so you wouldn't see the impact of t= hat patch yet.</div> <div class=3D"plain_line">=C2=A0</div> <blockquote type=3D"cite" class=3D"cite2"> <div class=3D"plain_line"> However, I also tried to remove some datastores, = which failed with</div> <div class=3D"plain_line"> timeouts. PBS even stopped authenticating (so pr= obably just working) all</div> <div class=3D"plain_line"> together for about 10 seconds, which was an unpl= easant surprise.</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> So looking into that further, I noticed the foll= owing logging:</div> <div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET</div> <div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client</div> <div class=3D"plain_line"> [::ffff]:42104] Unable to acquire lock</div> <div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error</div> <div class=3D"plain_line"> 4)</div> <div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET</div> <div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client</div> <div class=3D"plain_line"> [::ffff]:42144] Unable to acquire lock</div> <div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error</div> <div class=3D"plain_line"> 4)</div> <div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET</div> <div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client</div> <div class=3D"plain_line"> [::ffff]:47286] Unable to acquire lock</div> <div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error</div> <div class=3D"plain_line"> 4)</div> <div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET</div> <div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client</div> <div class=3D"plain_line"> [::ffff:]:45994] Unable to acquire lock</div> <div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error</div> <div class=3D"plain_line"> 4)</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> Which surprised me, since this is a =E2=80=99sta= tus=E2=80=99 call, which should not need</div> <div class=3D"plain_line"> locking of the datastore-config.</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> <a href=3D"https://git.proxmox.com/?p=3Dproxmox-= backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3Dc611f593624977defc4= 9d6e4de2ab8185cfe09e9;hb=3DHEAD#l687" class=3D"__cef_visited">https://git.p= roxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs= ;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l687</a></div> <div class=3D"plain_line"> does not lock the config, but</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> <a href=3D"https://git.proxmox.com/?p=3Dproxmox-= backup.git;a=3Dblob;f=3Dpbs-datastore/src/datastore.rs;h=3D0801b4bf6b25eaa6= f306c7b39ae2cfe81b4782e1;hb=3DHEAD#l204" class=3D"__cef_visited">https://gi= t.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-datastore/src/datast= ore.rs;h=3D0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=3DHEAD#l204</a></div= > <div class=3D"plain_line"> does.</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> So if I understand this correctly, every =E2=80= =99status=E2=80=99 call (30 per second in</div> <div class=3D"plain_line"> our case) locks the datastore-config exclusively= . And also, every time</div> <div class=3D"plain_line"> =E2=80=99status=E2=80=99 get called, the whole d= atastore-config gets loaded?</div> </blockquote> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">probably, there are some comments about that ther= e already, it might</div> <div class=3D"plain_line">make sense to open a bugzilla issue to discuss th= is further [1].</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">[1]: <a href=3D"https://bugzilla.proxmox.com/">ht= tps://bugzilla.proxmox.com/</a></div> <div class=3D"plain_line">=C2=A0</div> <blockquote type=3D"cite" class=3D"cite2"> <div class=3D"plain_line"> Is that something that could use some performanc= e tuning?</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> =E2=80=94</div> <div class=3D"plain_line"> Mark Schouten</div> <div class=3D"plain_line"> CTO, Tuxis B.V.</div> <div class=3D"plain_line"> +31 318 200208 / <a href=3D"mailto:mark@tuxis.nl= ">mark@tuxis.nl</a></div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> ------ Original Message ------</div> <div class=3D"plain_line"> From "Shannon Sterz" <<a href=3D"mailto:s.ste= rz@proxmox.com">s.sterz@proxmox.com</a>></div> <div class=3D"plain_line"> To "Mark Schouten" <<a href=3D"mailto:mark@tu= xis.nl">mark@tuxis.nl</a>></div> <div class=3D"plain_line"> Cc "Proxmox Backup Server development discussion= "</div> <div class=3D"plain_line"> <<a href=3D"mailto:pbs-devel@lists.proxmox.co= m">pbs-devel@lists.proxmox.com</a>></div> <div class=3D"plain_line"> Date 16/12/2024 12:51:47</div> <div class=3D"plain_line"> Subject Re: Re[2]: [pbs-devel] Authentication pe= rformance</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> >On Mon Dec 16, 2024 at 12:23 PM CET, Mark Sc= houten wrote:</div> <div class=3D"plain_line"> >> Hi,</div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> ></div> <div class=3D"plain_line"> >> >would you mind sharing either `aut= hkey.pub` or the output of the</div> <div class=3D"plain_line"> >> >following commands:</div> <div class=3D"plain_line"> >> ></div> <div class=3D"plain_line"> >> >head --lines=3D1 /etc/proxmox-back= up/authkey.key</div> <div class=3D"plain_line"> >> >cat /etc/proxmox-backup/authkey.ke= y | wc -l</div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> -----BEGIN RSA PRIVATE KEY-----</div> <div class=3D"plain_line"> >> 51</div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> So that is indeed the legacy method. W= e are going to upgrade our PBS=E2=80=99es</div> <div class=3D"plain_line"> >> on wednesday.</div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> ></div> <div class=3D"plain_line"> >> >The first should give the PEM head= er of the authkey whereas the second</div> <div class=3D"plain_line"> >> >provides the amount of lines that= the key takes up in the file. Both</div> <div class=3D"plain_line"> >> >give an indication whether you are = using the legacy RSA keys or newer</div> <div class=3D"plain_line"> >> >Ed25519 keys. The later should pro= vide more performance, security should</div> <div class=3D"plain_line"> >> >not be affected much by this chang= e. If the output of the commands look</div> <div class=3D"plain_line"> >> >like this:</div> <div class=3D"plain_line"> >> ></div> <div class=3D"plain_line"> >> >-----BEGIN PRIVATE KEY-----</div> <div class=3D"plain_line"> >> >3</div> <div class=3D"plain_line"> >> ></div> <div class=3D"plain_line"> >> >Then you are using the newer keys. = There currently isn't a recommended</div> <div class=3D"plain_line"> >> >way to upgrade the keys. However,= in theory you should be able to remove</div> <div class=3D"plain_line"> >> >the old keys, re-start PBS and it= should just generate keys in the new</div> <div class=3D"plain_line"> >> >format. Note that this will logout = anyone that is currently</div> <div class=3D"plain_line"> >> >authenticated and they'll have to= re-authenticate.</div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> Seems like a good moment to update t= hose keys as well.</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >Sure, just be aware that you have to manuall= y delete the key before</div> <div class=3D"plain_line"> >restarting the PBS. Upgrading alone won't af= fect the key. Ideally you'd</div> <div class=3D"plain_line"> >test this before rolling it out, if you can<= /div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >> >In general, tokens should still be = fater to authenticate so we'd</div> <div class=3D"plain_line"> >> >recommend that you try to get your = users to switch to token-based</div> <div class=3D"plain_line"> >> >authentication where possible. Imp= roving performance there is a bit</div> <div class=3D"plain_line"> >> >trickier though, as it often comes = with a security trade-off (in the</div> <div class=3D"plain_line"> >> >background we use yescrypt fo the= authentication there, that</div> <div class=3D"plain_line"> >> >delibaretely adds a work factor).= However, we may be able to improve</div> <div class=3D"plain_line"> >> >performance a bit via caching meth= ods or similar.</div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> Yes, that might help. I=E2=80=99m also = not sure if it actually is</div> <div class=3D"plain_line"> >> authentication, or if it is the datast= ore-call that the PVE-environments</div> <div class=3D"plain_line"> >> call. As you can see in your support i= ssue 3153557, it looks like some</div> <div class=3D"plain_line"> >> requests loop through all datastores,= before responding with a limited</div> <div class=3D"plain_line"> >> set of datastores.</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >I looked at that ticket and yes, that is pro= bably unrelated to</div> <div class=3D"plain_line"> >authentication.</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >> For instance (and I=E2=80=99m a comple= te noob wrt Rust) but if I understand</div> <div class=3D"plain_line"> >><a href=3D"https://git.proxmox.com/?p=3D= proxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3D11d2641b9ca= 2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD#l1315">https://git.proxmox.com/?p= =3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3D11d2641b= 9ca2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD#l1315</a></div> <div class=3D"plain_line"> >> correcly, PBS loops through all the da= tastores, checks mount-status and</div> <div class=3D"plain_line"> >> config, and only starts filtering at l= ine 1347. If I understand that</div> <div class=3D"plain_line"> >> correctly, in our case with over 1100= datastores, that might cause quite</div> <div class=3D"plain_line"> >> some load?</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >Possible, yes, that would depend on your con= figuration. Are all of these</div> <div class=3D"plain_line"> >datastores defined with a backing device? Be= cause if not, than this</div> <div class=3D"plain_line"> >should be fairly fast (as in, this should no= t actually touch the disks).</div> <div class=3D"plain_line"> >If they are, then yes this could be slow as= each store would trigger at</div> <div class=3D"plain_line"> >least 2 stat calls afaict.</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >In any case, it should be fine to move the `= mount_status` check after</div> <div class=3D"plain_line"> >the `if allowed || allow_id` check from what = i can tell. Not sure why</div> <div class=3D"plain_line"> >we'd need to check the mount_status for a da= tastore we won't include in</div> <div class=3D"plain_line"> >the resulsts anyway. Same goes for parsing t= he store config imo. Send a</div> <div class=3D"plain_line"> >patch for that [1].</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >[1]: <a href=3D"https://lore.proxmox.com/pbs= -devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u">https://lore.proxm= ox.com/pbs-devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u</a></div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> Thanks,</div> <div class=3D"plain_line"> >></div> <div class=3D"plain_line"> >> =E2=80=94</div> <div class=3D"plain_line"> >> Mark Schouten</div> <div class=3D"plain_line"> >> CTO, Tuxis B.V.</div> <div class=3D"plain_line"> >> +31 318 200208 / <a href=3D"mailto:mar= k@tuxis.nl">mark@tuxis.nl</a></div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> ></div> </blockquote> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">=C2=A0</div> </blockquote></div> </blockquote></div> </body></html> --------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7-- --===============6086177299285116599== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel --===============6086177299285116599==--