From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
	by lore.proxmox.com (Postfix) with ESMTPS id 173781FF16B
	for <inbox@lore.proxmox.com>; Thu,  6 Mar 2025 15:02:48 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id E01CE7EA3;
	Thu,  6 Mar 2025 15:02:42 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=tuxis.nl; s=mail;
 h=from:reply-to:subject:date:message-id:to:cc:mime-version:content-type:
 in-reply-to:references;
 bh=P+of+ntNIcs9zLYbTUGMGXXBjWaR6b3/xxrPLNjLKWw=;
 b=OwnuHVvbl6Vqsym+D9tVrcqlytFXMkZZLEXfPHpPBDYJXQyjudQTJTAyNsONlMXXYs9PUOhw9QZnY
 oHfvdvZSUfbB6zMzoDYdzS7xJkx+O8qOFt3y9WmG/tFWr7cmltvNOgg8tqZlvh0Yww/lU6J3doGgnQ
 24IGIQs4/vaXIZM+zfWnC04/zoFvvWq0zbi0a+rQejRVmoELvfnDep1zMexCTcsNmbsln3knhPx2Ea
 lm0cphEmkUvPjKIv0WIu4KvQXgS+fkqx5M1sQ2NMT/chkbTroigqJxQtvURj3DBc7zJ+bA58J9oR+4
 a18kqdWh6fVqWlUTtPOa5gs3M5kftJQ==
X-Footer: dHV4aXMubmw=
From: "Mark Schouten" <mark@tuxis.nl>
To: "Shannon Sterz" <s.sterz@proxmox.com>
Date: Thu, 06 Mar 2025 13:32:15 +0000
Message-Id: <em372c8493-fc98-4f26-af91-658a1b05061e@fb5eb041.com>
In-Reply-To: <em808af812-c25f-4569-8e1c-231663ec413f@d1b4cc0c.com>
References: <ema6032dba-8585-4377-bec1-11a37159087c@192dede7.com>
 <D6D024F1JMI5.QDFXDKCQMUCJ@proxmox.com>
 <embeb48874-d400-4e69-ae0f-2cc56a39d592@93f95f61.com>
 <D6D3QC6Y5H4S.1QHYHPHXK6RVR@proxmox.com>
 <em2d4327d4-4607-458e-880d-9b6bfba58f0a@ef757d37.com>
 <D6GK5TIN2LD4.1AXEF95IHZITS@proxmox.com>
 <em808af812-c25f-4569-8e1c-231663ec413f@d1b4cc0c.com>
User-Agent: eM_Client/10.1.4828.0
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results:  0
 AWL -0.105 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DKIM_SIGNED               0.1 Message has a DKIM or DK signature,
 not necessarily valid
 DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
 DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's
 domain
 DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from
 domain DMARC_PASS               -0.1 DMARC pass policy
 HTML_MESSAGE            0.001 HTML included in message
 KAM_LOTSOFHASH           0.25 Emails with lots of hash-like gibberish
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Subject: Re: [pbs-devel] Authentication performance
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Mark Schouten <mark@tuxis.nl>,
 Proxmox Backup Server development discussion
 <pbs-devel@lists.proxmox.com>
Cc: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com>
Content-Type: multipart/mixed; boundary="===============6086177299285116599=="
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>

--===============6086177299285116599==
Content-Type: multipart/alternative;
 boundary="------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7"

--------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable

Hi,

Is there anything I can do to make this bug go up on the priority list?=20
:)

We=E2=80=99ve currently stopped supplying free accounts to the masses, beca=
use=20
we are pretty sure that it will cause more and more issues to our setup=20
if we keep creating more datastores and stuff.

Please let me know..

=E2=80=94
Mark Schouten
CTO, Tuxis B.V.
+31 318 200208 / mark@tuxis.nl


------ Original Message ------
>From "Mark Schouten" <mark@tuxis.nl>
To "Shannon Sterz" <s.sterz@proxmox.com>
Cc "Proxmox Backup Server development discussion"=20
<pbs-devel@lists.proxmox.com>
Date 06/01/2025 20:07:43
Subject Re[6]: [pbs-devel] Authentication performance

>https://bugzilla.proxmox.com/show_bug.cgi?id=3D6049 has been created for=
=20
>this.
>
>Thanks!
>
>=E2=80=94
>Mark Schouten
>CTO, Tuxis B.V.
>+31 318 200208 / mark@tuxis.nl
>
>
>------ Original Message ------
>>From "Shannon Sterz" <s.sterz@proxmox.com>
>To "Mark Schouten" <mark@tuxis.nl>
>Cc "Proxmox Backup Server development discussion"=20
><pbs-devel@lists.proxmox.com>
>Date 20/12/2024 14:22:18
>Subject Re: Re[4]: [pbs-devel] Authentication performance
>
>>On Thu Dec 19, 2024 at 10:56 AM CET, Mark Schouten wrote:
>>>  Hi,
>>>
>>>  We upgraded to 3.3 yesterday, not much gain to notice with regards to
>>>  the new version or the change in keying. It=E2=80=99s still (obvioulsy=
) pretty
>>>  busy.
>>
>>just be aware that the patch i linked to in my last mail has not been
>>packaged yet, so you wouldn't see the impact of that patch yet.
>>
>>>  However, I also tried to remove some datastores, which failed with
>>>  timeouts. PBS even stopped authenticating (so probably just working) a=
ll
>>>  together for about 10 seconds, which was an unpleasant surprise.
>>>
>>>  So looking into that further, I noticed the following logging:
>>>  Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET
>>>  /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client
>>>  [::ffff]:42104] Unable to acquire lock
>>>  "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err=
or
>>>  4)
>>>  Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET
>>>  /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client
>>>  [::ffff]:42144] Unable to acquire lock
>>>  "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err=
or
>>>  4)
>>>  Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET
>>>  /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client
>>>  [::ffff]:47286] Unable to acquire lock
>>>  "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err=
or
>>>  4)
>>>  Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET
>>>  /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client
>>>  [::ffff:]:45994] Unable to acquire lock
>>>  "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os err=
or
>>>  4)
>>>
>>>  Which surprised me, since this is a =E2=80=99status=E2=80=99 call, whi=
ch should not need
>>>  locking of the datastore-config.
>>>
>>>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/ad=
min/datastore.rs;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l68=
7
>>>  does not lock the config, but
>>>
>>>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-datasto=
re/src/datastore.rs;h=3D0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=3DHEAD#=
l204
>>>  does.
>>>
>>>  So if I understand this correctly, every =E2=80=99status=E2=80=99 call =
(30 per second in
>>>  our case) locks the datastore-config exclusively. And also, every time
>>>  =E2=80=99status=E2=80=99 get called, the whole datastore-config gets l=
oaded?
>>
>>probably, there are some comments about that there already, it might
>>make sense to open a bugzilla issue to discuss this further [1].
>>
>>[1]: https://bugzilla.proxmox.com/
>>
>>>  Is that something that could use some performance tuning?
>>>
>>>  =E2=80=94
>>>  Mark Schouten
>>>  CTO, Tuxis B.V.
>>>  +31 318 200208 / mark@tuxis.nl
>>>
>>>
>>>  ------ Original Message ------
>>>  From "Shannon Sterz" <s.sterz@proxmox.com>
>>>  To "Mark Schouten" <mark@tuxis.nl>
>>>  Cc "Proxmox Backup Server development discussion"
>>>  <pbs-devel@lists.proxmox.com>
>>>  Date 16/12/2024 12:51:47
>>>  Subject Re: Re[2]: [pbs-devel] Authentication performance
>>>
>>>  >On Mon Dec 16, 2024 at 12:23 PM CET, Mark Schouten wrote:
>>>  >>  Hi,
>>>  >>
>>>  >>  >
>>>  >>  >would you mind sharing either `authkey.pub` or the output of the
>>>  >>  >following commands:
>>>  >>  >
>>>  >>  >head --lines=3D1 /etc/proxmox-backup/authkey.key
>>>  >>  >cat /etc/proxmox-backup/authkey.key | wc -l
>>>  >>
>>>  >>  -----BEGIN RSA PRIVATE KEY-----
>>>  >>  51
>>>  >>
>>>  >>  So that is indeed the legacy method. We are going to upgrade our P=
BS=E2=80=99es
>>>  >>  on wednesday.
>>>  >>
>>>  >>  >
>>>  >>  >The first should give the PEM header of the authkey whereas the s=
econd
>>>  >>  >provides the amount of lines that the key takes up in the file. B=
oth
>>>  >>  >give an indication whether you are using the legacy RSA keys or n=
ewer
>>>  >>  >Ed25519 keys. The later should provide more performance, security =
should
>>>  >>  >not be affected much by this change. If the output of the command=
s look
>>>  >>  >like this:
>>>  >>  >
>>>  >>  >-----BEGIN PRIVATE KEY-----
>>>  >>  >3
>>>  >>  >
>>>  >>  >Then you are using the newer keys. There currently isn't a recomm=
ended
>>>  >>  >way to upgrade the keys. However, in theory you should be able to =
remove
>>>  >>  >the old keys, re-start PBS and it should just generate keys in th=
e new
>>>  >>  >format. Note that this will logout anyone that is currently
>>>  >>  >authenticated and they'll have to re-authenticate.
>>>  >>
>>>  >>    Seems like a good moment to update those keys as well.
>>>  >
>>>  >Sure, just be aware that you have to manually delete the key before
>>>  >restarting the PBS. Upgrading alone won't affect the key. Ideally you=
'd
>>>  >test this before rolling it out, if you can
>>>  >
>>>  >>  >In general, tokens should still be fater to authenticate so we'd
>>>  >>  >recommend that you try to get your users to switch to token-based
>>>  >>  >authentication where possible. Improving performance there is a b=
it
>>>  >>  >trickier though, as it often comes with a security trade-off (in=
 the
>>>  >>  >background we use yescrypt fo the authentication there, that
>>>  >>  >delibaretely adds a work factor). However, we may be able to impr=
ove
>>>  >>  >performance a bit via caching methods or similar.
>>>  >>
>>>  >>  Yes, that might help. I=E2=80=99m also not sure if it actually is
>>>  >>  authentication, or if it is the datastore-call that the PVE-enviro=
nments
>>>  >>  call. As you can see in your support issue 3153557, it looks like=
 some
>>>  >>  requests loop through all datastores, before responding with a lim=
ited
>>>  >>  set of datastores.
>>>  >
>>>  >I looked at that ticket and yes, that is probably unrelated to
>>>  >authentication.
>>>  >
>>>  >>  For instance (and I=E2=80=99m a complete noob wrt Rust) but if I u=
nderstand
>>>  >>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api=
2/admin/datastore.rs;h=3D11d2641b9ca2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD=
#l1315
>>>  >>  correcly, PBS loops through all the datastores, checks mount-statu=
s and
>>>  >>  config, and only starts filtering at line 1347. If I understand th=
at
>>>  >>  correctly, in our case with over 1100 datastores, that might cause =
quite
>>>  >>  some load?
>>>  >
>>>  >Possible, yes, that would depend on your configuration. Are all of th=
ese
>>>  >datastores defined with a backing device? Because if not, than this
>>>  >should be fairly fast (as in, this should not actually touch the disk=
s).
>>>  >If they are, then yes this could be slow as each store would trigger=
 at
>>>  >least 2 stat calls afaict.
>>>  >
>>>  >In any case, it should be fine to move the `mount_status` check after
>>>  >the `if allowed || allow_id` check from what i can tell. Not sure why
>>>  >we'd need to check the mount_status for a datastore we won't include=
 in
>>>  >the resulsts anyway. Same goes for parsing the store config imo. Send =
a
>>>  >patch for that [1].
>>>  >
>>>  >[1]: https://lore.proxmox.com/pbs-devel/20241216115044.208595-1-s.ste=
rz@proxmox.com/T/#u
>>>  >
>>>  >>
>>>  >>
>>>  >>  Thanks,
>>>  >>
>>>  >>  =E2=80=94
>>>  >>    Mark Schouten
>>>  >>  CTO, Tuxis B.V.
>>>  >>  +31 318 200208 / mark@tuxis.nl
>>>  >
>>>  >
>>
>>
--------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head>

<style type=3D"text/css"><!--#x3ccf03dbbf4f4dc blockquote.cite
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p=
x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb=
(204, 204, 204);}
#x3ccf03dbbf4f4dc blockquote.cite2
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p=
x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb=
(204, 204, 204); margin-top: 3px; padding-top: 0px;}
#x3ccf03dbbf4f4dc
{font-family: Helvetica; font-size: 9pt;}
--></style><style id=3D"css_styles" type=3D"text/css"><!--blockquote.cite { =
margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px=
; border-left: 1px solid #cccccc }
blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px;=
 padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding=
-top: 0px; }
a img { border: 0px; }
table { border-collapse: collapse; }
li[style=3D'text-align: center;'], li[style=3D'text-align: center; '], li[s=
tyle=3D'text-align: right;'], li[style=3D'text-align: right; '] {  list-sty=
le-position: inside;}
body { font-family: Helvetica; font-size: 9pt; }
.quote { margin-left: 1em; margin-right: 1em; border-left: 5px #ebebeb soli=
d; padding-left: 0.3em; }
a.em-mention[href] { text-decoration: none; color: inherit; border-radius:=
 3px; padding-left: 2px; padding-right: 2px; background-color: #e2e2e2; }
--></style></head>
<body style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; line-br=
eak: after-white-space;"><div>Hi,</div><div><br /></div><div>Is there anyth=
ing I can do to make this bug go up on the priority list?=C2=A0<span style=
=3D"color: rgb(210, 210, 210); font-size: 9pt;">:)</span></div><div><span><=
br /></span></div><div>We=E2=80=99ve currently stopped supplying free accou=
nts to the masses, because we are pretty sure that it will cause more and m=
ore issues to our setup if we keep creating more datastores and stuff.</div=
><div><br /></div><div>Please let me know..</div><div><br /></div><div id=
=3D"signature_old" style=3D"clear:both"><div style=3D"margin: 0px; padding: =
0px; box-sizing: content-box;">=E2=80=94=C2=A0</div><div style=3D"margin:=
 0px; padding: 0px; box-sizing: content-box;">Mark Schouten</div><div style=
=3D"margin: 0px; padding: 0px; box-sizing: content-box;">CTO, Tuxis B.V.</d=
iv><div style=3D"margin: 0px; padding: 0px; box-sizing: content-box;">+31 3=
18 200208 /=C2=A0mark@tuxis.nl</div></div><div><br /></div>
<div x-em-replyforwardheader=3D""><br /></div>
<div>
<div>------ Original Message ------</div>
<div>From "Mark Schouten" &lt;<a href=3D"mailto:mark@tuxis.nl">mark@tuxis.n=
l</a>&gt;</div>
<div>To "Shannon Sterz" &lt;<a href=3D"mailto:s.sterz@proxmox.com">s.sterz@=
proxmox.com</a>&gt;</div>
<div>Cc "Proxmox Backup Server development discussion" &lt;<a href=3D"mailt=
o:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>&gt;</div>
<div>Date 06/01/2025 20:07:43</div>
<div>Subject Re[6]: [pbs-devel] Authentication performance</div></div><div=
 x-em-quote=3D""><br /></div>
<div id=3D"x3ccf03dbbf4f4dc" style=3D"overflow-wrap: break-word; -webkit-nb=
sp-mode: space; line-break: after-white-space;"><blockquote cite=3D"em808af=
812-c25f-4569-8e1c-231663ec413f@d1b4cc0c.com" type=3D"cite" class=3D"cite2"=
>
<div><a href=3D"https://bugzilla.proxmox.com/show_bug.cgi?id=3D6049">https:=
//bugzilla.proxmox.com/show_bug.cgi?id=3D6049</a>=C2=A0has been created for =
this.</div><div><br /></div><div>Thanks!</div>
<div><br /></div><div id=3D"signature_old" style=3D"clear:both"><div style=
=3D"margin: 0px; padding: 0px; box-sizing: content-box;">=E2=80=94=C2=A0</d=
iv><div style=3D"margin: 0px; padding: 0px; box-sizing: content-box;">Mark=
 Schouten</div><div style=3D"margin: 0px; padding: 0px; box-sizing: content-=
box;">CTO, Tuxis B.V.</div><div style=3D"margin: 0px; padding: 0px; box-siz=
ing: content-box;">+31 318 200208 /=C2=A0mark@tuxis.nl</div></div><div><br=
 /></div>
<div x-em-replyforwardheader=3D""><br /></div>
<div>
<div>------ Original Message ------</div>
<div>From "Shannon Sterz" &lt;<a href=3D"mailto:s.sterz@proxmox.com">s.ster=
z@proxmox.com</a>&gt;</div>
<div>To "Mark Schouten" &lt;<a href=3D"mailto:mark@tuxis.nl">mark@tuxis.nl<=
/a>&gt;</div>
<div>Cc "Proxmox Backup Server development discussion" &lt;<a href=3D"mailt=
o:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>&gt;</div>
<div>Date 20/12/2024 14:22:18</div>
<div>Subject Re: Re[4]: [pbs-devel] Authentication performance</div></div><=
div x-em-quote=3D""><br /></div>
<div id=3D"xd7a8ca7f59be40f" class=3D"plain"><blockquote cite=3D"D6GK5TIN2L=
D4.1AXEF95IHZITS@proxmox.com" type=3D"cite" class=3D"cite2">

<div class=3D"plain_line">On Thu Dec 19, 2024 at 10:56 AM CET, Mark Schoute=
n wrote:</div>
<blockquote type=3D"cite" class=3D"cite">
<div class=3D"plain_line"> Hi,</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> We upgraded to 3.3 yesterday, not much gain to n=
otice with regards to</div>
<div class=3D"plain_line"> the new version or the change in keying. It=E2=
=80=99s still (obvioulsy) pretty</div>
<div class=3D"plain_line"> busy.</div>
</blockquote>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line">just be aware that the patch i linked to in my la=
st mail has not been</div>
<div class=3D"plain_line">packaged yet, so you wouldn't see the impact of t=
hat patch yet.</div>
<div class=3D"plain_line">=C2=A0</div>
<blockquote type=3D"cite" class=3D"cite2">
<div class=3D"plain_line"> However, I also tried to remove some datastores, =
which failed with</div>
<div class=3D"plain_line"> timeouts. PBS even stopped authenticating (so pr=
obably just working) all</div>
<div class=3D"plain_line"> together for about 10 seconds, which was an unpl=
easant surprise.</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> So looking into that further, I noticed the foll=
owing logging:</div>
<div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914=
3]: GET</div>
<div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba=
d Request: [client</div>
<div class=3D"plain_line"> [::ffff]:42104] Unable to acquire lock</div>
<div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt=
ed system call (os error</div>
<div class=3D"plain_line"> 4)</div>
<div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914=
3]: GET</div>
<div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba=
d Request: [client</div>
<div class=3D"plain_line"> [::ffff]:42144] Unable to acquire lock</div>
<div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt=
ed system call (os error</div>
<div class=3D"plain_line"> 4)</div>
<div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914=
3]: GET</div>
<div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba=
d Request: [client</div>
<div class=3D"plain_line"> [::ffff]:47286] Unable to acquire lock</div>
<div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt=
ed system call (os error</div>
<div class=3D"plain_line"> 4)</div>
<div class=3D"plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[3914=
3]: GET</div>
<div class=3D"plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Ba=
d Request: [client</div>
<div class=3D"plain_line"> [::ffff:]:45994] Unable to acquire lock</div>
<div class=3D"plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupt=
ed system call (os error</div>
<div class=3D"plain_line"> 4)</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> Which surprised me, since this is a =E2=80=99sta=
tus=E2=80=99 call, which should not need</div>
<div class=3D"plain_line"> locking of the datastore-config.</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> <a href=3D"https://git.proxmox.com/?p=3Dproxmox-=
backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3Dc611f593624977defc4=
9d6e4de2ab8185cfe09e9;hb=3DHEAD#l687" class=3D"__cef_visited">https://git.p=
roxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs=
;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l687</a></div>
<div class=3D"plain_line"> does not lock the config, but</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> <a href=3D"https://git.proxmox.com/?p=3Dproxmox-=
backup.git;a=3Dblob;f=3Dpbs-datastore/src/datastore.rs;h=3D0801b4bf6b25eaa6=
f306c7b39ae2cfe81b4782e1;hb=3DHEAD#l204" class=3D"__cef_visited">https://gi=
t.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-datastore/src/datast=
ore.rs;h=3D0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=3DHEAD#l204</a></div=
>
<div class=3D"plain_line"> does.</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> So if I understand this correctly, every =E2=80=
=99status=E2=80=99 call (30 per second in</div>
<div class=3D"plain_line"> our case) locks the datastore-config exclusively=
. And also, every time</div>
<div class=3D"plain_line"> =E2=80=99status=E2=80=99 get called, the whole d=
atastore-config gets loaded?</div>
</blockquote>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line">probably, there are some comments about that ther=
e already, it might</div>
<div class=3D"plain_line">make sense to open a bugzilla issue to discuss th=
is further [1].</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line">[1]: <a href=3D"https://bugzilla.proxmox.com/">ht=
tps://bugzilla.proxmox.com/</a></div>
<div class=3D"plain_line">=C2=A0</div>
<blockquote type=3D"cite" class=3D"cite2">
<div class=3D"plain_line"> Is that something that could use some performanc=
e tuning?</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> =E2=80=94</div>
<div class=3D"plain_line"> Mark Schouten</div>
<div class=3D"plain_line"> CTO, Tuxis B.V.</div>
<div class=3D"plain_line"> +31 318 200208 / <a href=3D"mailto:mark@tuxis.nl=
">mark@tuxis.nl</a></div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> ------ Original Message ------</div>
<div class=3D"plain_line"> From "Shannon Sterz" &lt;<a href=3D"mailto:s.ste=
rz@proxmox.com">s.sterz@proxmox.com</a>&gt;</div>
<div class=3D"plain_line"> To "Mark Schouten" &lt;<a href=3D"mailto:mark@tu=
xis.nl">mark@tuxis.nl</a>&gt;</div>
<div class=3D"plain_line"> Cc "Proxmox Backup Server development discussion=
"</div>
<div class=3D"plain_line"> &lt;<a href=3D"mailto:pbs-devel@lists.proxmox.co=
m">pbs-devel@lists.proxmox.com</a>&gt;</div>
<div class=3D"plain_line"> Date 16/12/2024 12:51:47</div>
<div class=3D"plain_line"> Subject Re: Re[2]: [pbs-devel] Authentication pe=
rformance</div>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line"> &gt;On Mon Dec 16, 2024 at 12:23 PM CET, Mark Sc=
houten wrote:</div>
<div class=3D"plain_line"> &gt;&gt;  Hi,</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;would you mind sharing either `aut=
hkey.pub` or the output of the</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;following commands:</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;head --lines=3D1 /etc/proxmox-back=
up/authkey.key</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;cat /etc/proxmox-backup/authkey.ke=
y | wc -l</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;  -----BEGIN RSA PRIVATE KEY-----</div>
<div class=3D"plain_line"> &gt;&gt;  51</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;  So that is indeed the legacy method. W=
e are going to upgrade our PBS=E2=80=99es</div>
<div class=3D"plain_line"> &gt;&gt;  on wednesday.</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;The first should give the PEM head=
er of the authkey whereas the second</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;provides the amount of lines that=
 the key takes up in the file. Both</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;give an indication whether you are =
using the legacy RSA keys or newer</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;Ed25519 keys. The later should pro=
vide more performance, security should</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;not be affected much by this chang=
e. If the output of the commands look</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;like this:</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;-----BEGIN PRIVATE KEY-----</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;3</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;Then you are using the newer keys. =
There currently isn't a recommended</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;way to upgrade the keys. However,=
 in theory you should be able to remove</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;the old keys, re-start PBS and it=
 should just generate keys in the new</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;format. Note that this will logout =
anyone that is currently</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;authenticated and they'll have to=
 re-authenticate.</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;    Seems like a good moment to update t=
hose keys as well.</div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;Sure, just be aware that you have to manuall=
y delete the key before</div>
<div class=3D"plain_line"> &gt;restarting the PBS. Upgrading alone won't af=
fect the key. Ideally you'd</div>
<div class=3D"plain_line"> &gt;test this before rolling it out, if you can<=
/div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;In general, tokens should still be =
fater to authenticate so we'd</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;recommend that you try to get your =
users to switch to token-based</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;authentication where possible. Imp=
roving performance there is a bit</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;trickier though, as it often comes =
with a security trade-off (in the</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;background we use yescrypt fo the=
 authentication there, that</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;delibaretely adds a work factor).=
 However, we may be able to improve</div>
<div class=3D"plain_line"> &gt;&gt;  &gt;performance a bit via caching meth=
ods or similar.</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;  Yes, that might help. I=E2=80=99m also =
not sure if it actually is</div>
<div class=3D"plain_line"> &gt;&gt;  authentication, or if it is the datast=
ore-call that the PVE-environments</div>
<div class=3D"plain_line"> &gt;&gt;  call. As you can see in your support i=
ssue 3153557, it looks like some</div>
<div class=3D"plain_line"> &gt;&gt;  requests loop through all datastores,=
 before responding with a limited</div>
<div class=3D"plain_line"> &gt;&gt;  set of datastores.</div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;I looked at that ticket and yes, that is pro=
bably unrelated to</div>
<div class=3D"plain_line"> &gt;authentication.</div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;&gt;  For instance (and I=E2=80=99m a comple=
te noob wrt Rust) but if I understand</div>
<div class=3D"plain_line"> &gt;&gt;<a href=3D"https://git.proxmox.com/?p=3D=
proxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3D11d2641b9ca=
2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD#l1315">https://git.proxmox.com/?p=
=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3D11d2641b=
9ca2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD#l1315</a></div>
<div class=3D"plain_line"> &gt;&gt;  correcly, PBS loops through all the da=
tastores, checks mount-status and</div>
<div class=3D"plain_line"> &gt;&gt;  config, and only starts filtering at l=
ine 1347. If I understand that</div>
<div class=3D"plain_line"> &gt;&gt;  correctly, in our case with over 1100=
 datastores, that might cause quite</div>
<div class=3D"plain_line"> &gt;&gt;  some load?</div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;Possible, yes, that would depend on your con=
figuration. Are all of these</div>
<div class=3D"plain_line"> &gt;datastores defined with a backing device? Be=
cause if not, than this</div>
<div class=3D"plain_line"> &gt;should be fairly fast (as in, this should no=
t actually touch the disks).</div>
<div class=3D"plain_line"> &gt;If they are, then yes this could be slow as=
 each store would trigger at</div>
<div class=3D"plain_line"> &gt;least 2 stat calls afaict.</div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;In any case, it should be fine to move the `=
mount_status` check after</div>
<div class=3D"plain_line"> &gt;the `if allowed || allow_id` check from what =
i can tell. Not sure why</div>
<div class=3D"plain_line"> &gt;we'd need to check the mount_status for a da=
tastore we won't include in</div>
<div class=3D"plain_line"> &gt;the resulsts anyway. Same goes for parsing t=
he store config imo. Send a</div>
<div class=3D"plain_line"> &gt;patch for that [1].</div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;[1]: <a href=3D"https://lore.proxmox.com/pbs=
-devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u">https://lore.proxm=
ox.com/pbs-devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u</a></div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;  Thanks,</div>
<div class=3D"plain_line"> &gt;&gt;</div>
<div class=3D"plain_line"> &gt;&gt;  =E2=80=94</div>
<div class=3D"plain_line"> &gt;&gt;    Mark Schouten</div>
<div class=3D"plain_line"> &gt;&gt;  CTO, Tuxis B.V.</div>
<div class=3D"plain_line"> &gt;&gt;  +31 318 200208 / <a href=3D"mailto:mar=
k@tuxis.nl">mark@tuxis.nl</a></div>
<div class=3D"plain_line"> &gt;</div>
<div class=3D"plain_line"> &gt;</div>
</blockquote>
<div class=3D"plain_line">=C2=A0</div>
<div class=3D"plain_line">=C2=A0</div>
</blockquote></div>


</blockquote></div>
</body></html>
--------=_MBFD471F7D-51B8-4E03-BC7A-CCE41EE123C7--




--===============6086177299285116599==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

--===============6086177299285116599==--