From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id D8A4B1FF16E for ; Mon, 6 Jan 2025 20:08:11 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 326D91D369; Mon, 6 Jan 2025 20:08:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=tuxis.nl; s=mail; h=from:reply-to:subject:date:message-id:to:cc:mime-version:content-type: in-reply-to:references; bh=cRqiXe2kc2Jom0e6E/YotBYlr+5MsS5qtL/b375U/IE=; b=EyMhnDrGHSGMbYmRQfPZl7EAEdFaPTv+yjeOvjW/uJVtPXtV2TbouOisUqokkP38kYbtOPOwQay2w 7BS0AjmvWXDbSVR2ptjkGFaryEIYBuCRBpCYeHqe56UQu4EdIj/PKROBt9dqIZ0F11To74ZTZyul4l tMSUEHbHM7QG92o4ntVUBivCm3CBN0bxOnByV57siPI4IpU4/T3MbgX8WN5vpIB+dAly6QsLGAZLXv IACtdIRRF/RlD1fpKpEwSiRE4NL/mHHHos4Av1csNfJM1WpbpmlGLoJEHisHifILmT44KT9UnDbfYu 67urATywC64AKjrooEM+sJbLr2nzbkw== X-Footer: dHV4aXMubmw= From: "Mark Schouten" To: "Shannon Sterz" Date: Mon, 06 Jan 2025 19:07:43 +0000 Message-Id: In-Reply-To: References: User-Agent: eM_Client/10.1.4828.0 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.108 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy HTML_MESSAGE 0.001 HTML included in message KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] Authentication performance X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Mark Schouten , Proxmox Backup Server development discussion Cc: Proxmox Backup Server development discussion Content-Type: multipart/mixed; boundary="===============2381593319938800892==" Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" --===============2381593319938800892== Content-Type: multipart/alternative; boundary="------=_MB1AC0B637-F58C-475C-9DC1-7B1A6ACD3A66" --------=_MB1AC0B637-F58C-475C-9DC1-7B1A6ACD3A66 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable https://bugzilla.proxmox.com/show_bug.cgi?id=3D6049 has been created for=20 this. Thanks! =E2=80=94 Mark Schouten CTO, Tuxis B.V. +31 318 200208 / mark@tuxis.nl ------ Original Message ------ >From "Shannon Sterz" To "Mark Schouten" Cc "Proxmox Backup Server development discussion"=20 Date 20/12/2024 14:22:18 Subject Re: Re[4]: [pbs-devel] Authentication performance >On Thu Dec 19, 2024 at 10:56 AM CET, Mark Schouten wrote: >> Hi, >> >> We upgraded to 3.3 yesterday, not much gain to notice with regards to >> the new version or the change in keying. It=E2=80=99s still (obvioulsy) = pretty >> busy. > >just be aware that the patch i linked to in my last mail has not been >packaged yet, so you wouldn't see the impact of that patch yet. > >> However, I also tried to remove some datastores, which failed with >> timeouts. PBS even stopped authenticating (so probably just working) al= l >> together for about 10 seconds, which was an unpleasant surprise. >> >> So looking into that further, I noticed the following logging: >> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >> [::ffff]:42104] Unable to acquire lock >> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os erro= r >> 4) >> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >> [::ffff]:42144] Unable to acquire lock >> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os erro= r >> 4) >> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >> [::ffff]:47286] Unable to acquire lock >> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os erro= r >> 4) >> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET >> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client >> [::ffff:]:45994] Unable to acquire lock >> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os erro= r >> 4) >> >> Which surprised me, since this is a =E2=80=99status=E2=80=99 call, whic= h should not need >> locking of the datastore-config. >> >>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/adm= in/datastore.rs;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l687 >> does not lock the config, but >> >>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-datastor= e/src/datastore.rs;h=3D0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=3DHEAD#l= 204 >> does. >> >> So if I understand this correctly, every =E2=80=99status=E2=80=99 call= (30 per second in >> our case) locks the datastore-config exclusively. And also, every time >> =E2=80=99status=E2=80=99 get called, the whole datastore-config gets lo= aded? > >probably, there are some comments about that there already, it might >make sense to open a bugzilla issue to discuss this further [1]. > >[1]: https://bugzilla.proxmox.com/ > >> Is that something that could use some performance tuning? >> >> =E2=80=94 >> Mark Schouten >> CTO, Tuxis B.V. >> +31 318 200208 / mark@tuxis.nl >> >> >> ------ Original Message ------ >> From "Shannon Sterz" >> To "Mark Schouten" >> Cc "Proxmox Backup Server development discussion" >> >> Date 16/12/2024 12:51:47 >> Subject Re: Re[2]: [pbs-devel] Authentication performance >> >> >On Mon Dec 16, 2024 at 12:23 PM CET, Mark Schouten wrote: >> >> Hi, >> >> >> >> > >> >> >would you mind sharing either `authkey.pub` or the output of the >> >> >following commands: >> >> > >> >> >head --lines=3D1 /etc/proxmox-backup/authkey.key >> >> >cat /etc/proxmox-backup/authkey.key | wc -l >> >> >> >> -----BEGIN RSA PRIVATE KEY----- >> >> 51 >> >> >> >> So that is indeed the legacy method. We are going to upgrade our PB= S=E2=80=99es >> >> on wednesday. >> >> >> >> > >> >> >The first should give the PEM header of the authkey whereas the se= cond >> >> >provides the amount of lines that the key takes up in the file. Bo= th >> >> >give an indication whether you are using the legacy RSA keys or ne= wer >> >> >Ed25519 keys. The later should provide more performance, security= should >> >> >not be affected much by this change. If the output of the commands = look >> >> >like this: >> >> > >> >> >-----BEGIN PRIVATE KEY----- >> >> >3 >> >> > >> >> >Then you are using the newer keys. There currently isn't a recomme= nded >> >> >way to upgrade the keys. However, in theory you should be able to= remove >> >> >the old keys, re-start PBS and it should just generate keys in the = new >> >> >format. Note that this will logout anyone that is currently >> >> >authenticated and they'll have to re-authenticate. >> >> >> >> Seems like a good moment to update those keys as well. >> > >> >Sure, just be aware that you have to manually delete the key before >> >restarting the PBS. Upgrading alone won't affect the key. Ideally you'= d >> >test this before rolling it out, if you can >> > >> >> >In general, tokens should still be fater to authenticate so we'd >> >> >recommend that you try to get your users to switch to token-based >> >> >authentication where possible. Improving performance there is a bi= t >> >> >trickier though, as it often comes with a security trade-off (in t= he >> >> >background we use yescrypt fo the authentication there, that >> >> >delibaretely adds a work factor). However, we may be able to impro= ve >> >> >performance a bit via caching methods or similar. >> >> >> >> Yes, that might help. I=E2=80=99m also not sure if it actually is >> >> authentication, or if it is the datastore-call that the PVE-environ= ments >> >> call. As you can see in your support issue 3153557, it looks like s= ome >> >> requests loop through all datastores, before responding with a limi= ted >> >> set of datastores. >> > >> >I looked at that ticket and yes, that is probably unrelated to >> >authentication. >> > >> >> For instance (and I=E2=80=99m a complete noob wrt Rust) but if I un= derstand >> >>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2= /admin/datastore.rs;h=3D11d2641b9ca2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD#= l1315 >> >> correcly, PBS loops through all the datastores, checks mount-status = and >> >> config, and only starts filtering at line 1347. If I understand tha= t >> >> correctly, in our case with over 1100 datastores, that might cause= quite >> >> some load? >> > >> >Possible, yes, that would depend on your configuration. Are all of the= se >> >datastores defined with a backing device? Because if not, than this >> >should be fairly fast (as in, this should not actually touch the disks= ). >> >If they are, then yes this could be slow as each store would trigger a= t >> >least 2 stat calls afaict. >> > >> >In any case, it should be fine to move the `mount_status` check after >> >the `if allowed || allow_id` check from what i can tell. Not sure why >> >we'd need to check the mount_status for a datastore we won't include i= n >> >the resulsts anyway. Same goes for parsing the store config imo. Send= a >> >patch for that [1]. >> > >> >[1]: https://lore.proxmox.com/pbs-devel/20241216115044.208595-1-s.ster= z@proxmox.com/T/#u >> > >> >> >> >> >> >> Thanks, >> >> >> >> =E2=80=94 >> >> Mark Schouten >> >> CTO, Tuxis B.V. >> >> +31 318 200208 / mark@tuxis.nl >> > >> > > > --------=_MB1AC0B637-F58C-475C-9DC1-7B1A6ACD3A66 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
https://bugzilla.proxmox.com/show_bug.cgi?id=3D6049= =C2=A0has been created for this.

Thanks!

=E2=80=94=C2=A0
Mark= Schouten
CTO, Tuxis B.V.
+31 318 200208 /=C2=A0mark@tuxis.nl

------ Original Message ------
From "Shannon Sterz" <s.ster= z@proxmox.com>
To "Mark Schouten" <mark@tuxis.nl<= /a>>
Date 20/12/2024 14:22:18
Subject Re: Re[4]: [pbs-devel] Authentication performance
<= div x-em-quote=3D"">
On Thu Dec 19, 2024 at 10:56 AM CET, Mark Schoute= n wrote:
Hi,
=C2=A0
We upgraded to 3.3 yesterday, not much gain to n= otice with regards to
the new version or the change in keying. It=E2= =80=99s still (obvioulsy) pretty
busy.
=C2=A0
just be aware that the patch i linked to in my la= st mail has not been
packaged yet, so you wouldn't see the impact of t= hat patch yet.
=C2=A0
However, I also tried to remove some datastores, = which failed with
timeouts. PBS even stopped authenticating (so pr= obably just working) all
together for about 10 seconds, which was an unpl= easant surprise.
=C2=A0
So looking into that further, I noticed the foll= owing logging:
Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET
/api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client
[::ffff]:42104] Unable to acquire lock
"/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error
4)
Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET
/api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client
[::ffff]:42144] Unable to acquire lock
"/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error
4)
Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET
/api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client
[::ffff]:47286] Unable to acquire lock
"/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error
4)
Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914= 3]: GET
/api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client
[::ffff:]:45994] Unable to acquire lock
"/etc/proxmox-backup/.datastore.lck" - Interrupt= ed system call (os error
4)
=C2=A0
Which surprised me, since this is a =E2=80=99sta= tus=E2=80=99 call, which should not need
locking of the datastore-config.
=C2=A0
https://git.p= roxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs= ;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l687
does not lock the config, but
=C2=A0
https://gi= t.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-datastore/src/datast= ore.rs;h=3D0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=3DHEAD#l204
does.
=C2=A0
So if I understand this correctly, every =E2=80= =99status=E2=80=99 call (30 per second in
our case) locks the datastore-config exclusively= . And also, every time
=E2=80=99status=E2=80=99 get called, the whole d= atastore-config gets loaded?
=C2=A0
probably, there are some comments about that ther= e already, it might
make sense to open a bugzilla issue to discuss th= is further [1].
=C2=A0
[1]: ht= tps://bugzilla.proxmox.com/
=C2=A0
Is that something that could use some performanc= e tuning?
=C2=A0
=E2=80=94
Mark Schouten
CTO, Tuxis B.V.
+31 318 200208 / mark@tuxis.nl
=C2=A0
=C2=A0
------ Original Message ------
From "Shannon Sterz" <s.sterz@proxmox.com>
To "Mark Schouten" <mark@tuxis.nl>
Cc "Proxmox Backup Server development discussion= "
<pbs-devel@lists.proxmox.com>
Date 16/12/2024 12:51:47
Subject Re: Re[2]: [pbs-devel] Authentication pe= rformance
=C2=A0
>On Mon Dec 16, 2024 at 12:23 PM CET, Mark Sc= houten wrote:
>> Hi,
>>
>> >
>> >would you mind sharing either `aut= hkey.pub` or the output of the
>> >following commands:
>> >
>> >head --lines=3D1 /etc/proxmox-back= up/authkey.key
>> >cat /etc/proxmox-backup/authkey.ke= y | wc -l
>>
>> -----BEGIN RSA PRIVATE KEY-----
>> 51
>>
>> So that is indeed the legacy method. W= e are going to upgrade our PBS=E2=80=99es
>> on wednesday.
>>
>> >
>> >The first should give the PEM head= er of the authkey whereas the second
>> >provides the amount of lines that= the key takes up in the file. Both
>> >give an indication whether you are = using the legacy RSA keys or newer
>> >Ed25519 keys. The later should pro= vide more performance, security should
>> >not be affected much by this chang= e. If the output of the commands look
>> >like this:
>> >
>> >-----BEGIN PRIVATE KEY-----
>> >3
>> >
>> >Then you are using the newer keys. = There currently isn't a recommended
>> >way to upgrade the keys. However,= in theory you should be able to remove
>> >the old keys, re-start PBS and it= should just generate keys in the new
>> >format. Note that this will logout = anyone that is currently
>> >authenticated and they'll have to= re-authenticate.
>>
>> Seems like a good moment to update t= hose keys as well.
>
>Sure, just be aware that you have to manuall= y delete the key before
>restarting the PBS. Upgrading alone won't af= fect the key. Ideally you'd
>test this before rolling it out, if you can<= /div>
>
>> >In general, tokens should still be = fater to authenticate so we'd
>> >recommend that you try to get your = users to switch to token-based
>> >authentication where possible. Imp= roving performance there is a bit
>> >trickier though, as it often comes = with a security trade-off (in the
>> >background we use yescrypt fo the= authentication there, that
>> >delibaretely adds a work factor).= However, we may be able to improve
>> >performance a bit via caching meth= ods or similar.
>>
>> Yes, that might help. I=E2=80=99m also = not sure if it actually is
>> authentication, or if it is the datast= ore-call that the PVE-environments
>> call. As you can see in your support i= ssue 3153557, it looks like some
>> requests loop through all datastores,= before responding with a limited
>> set of datastores.
>
>I looked at that ticket and yes, that is pro= bably unrelated to
>authentication.
>
>> For instance (and I=E2=80=99m a comple= te noob wrt Rust) but if I understand
>>https://git.proxmox.com/?p= =3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3D11d2641b= 9ca2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD#l1315
>> correcly, PBS loops through all the da= tastores, checks mount-status and
>> config, and only starts filtering at l= ine 1347. If I understand that
>> correctly, in our case with over 1100= datastores, that might cause quite
>> some load?
>
>Possible, yes, that would depend on your con= figuration. Are all of these
>datastores defined with a backing device? Be= cause if not, than this
>should be fairly fast (as in, this should no= t actually touch the disks).
>If they are, then yes this could be slow as= each store would trigger at
>least 2 stat calls afaict.
>
>In any case, it should be fine to move the `= mount_status` check after
>the `if allowed || allow_id` check from what = i can tell. Not sure why
>we'd need to check the mount_status for a da= tastore we won't include in
>the resulsts anyway. Same goes for parsing t= he store config imo. Send a
>patch for that [1].
>
>[1]: https://lore.proxm= ox.com/pbs-devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u
>
>>
>>
>> Thanks,
>>
>> =E2=80=94
>> Mark Schouten
>> CTO, Tuxis B.V.
>> +31 318 200208 / mark@tuxis.nl
>
>
=C2=A0
=C2=A0
--------=_MB1AC0B637-F58C-475C-9DC1-7B1A6ACD3A66-- --===============2381593319938800892== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel --===============2381593319938800892==--