<html><head>
<style type="text/css"><!--#x3ccf03dbbf4f4dc blockquote.cite
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0px; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204);}
#x3ccf03dbbf4f4dc blockquote.cite2
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0px; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); margin-top: 3px; padding-top: 0px;}
#x3ccf03dbbf4f4dc
{font-family: Helvetica; font-size: 9pt;}
--></style><style id="css_styles" type="text/css"><!--blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc }
blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; }
a img { border: 0px; }
table { border-collapse: collapse; }
li[style='text-align: center;'], li[style='text-align: center; '], li[style='text-align: right;'], li[style='text-align: right; '] { list-style-position: inside;}
body { font-family: Helvetica; font-size: 9pt; }
.quote { margin-left: 1em; margin-right: 1em; border-left: 5px #ebebeb solid; padding-left: 0.3em; }
a.em-mention[href] { text-decoration: none; color: inherit; border-radius: 3px; padding-left: 2px; padding-right: 2px; background-color: #e2e2e2; }
--></style></head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>Hi,</div><div><br /></div><div>Is there anything I can do to make this bug go up on the priority list? <span style="color: rgb(210, 210, 210); font-size: 9pt;">:)</span></div><div><span><br /></span></div><div>We’ve currently stopped supplying free accounts to the masses, because we are pretty sure that it will cause more and more issues to our setup if we keep creating more datastores and stuff.</div><div><br /></div><div>Please let me know..</div><div><br /></div><div id="signature_old" style="clear:both"><div style="margin: 0px; padding: 0px; box-sizing: content-box;">— </div><div style="margin: 0px; padding: 0px; box-sizing: content-box;">Mark Schouten</div><div style="margin: 0px; padding: 0px; box-sizing: content-box;">CTO, Tuxis B.V.</div><div style="margin: 0px; padding: 0px; box-sizing: content-box;">+31 318 200208 / mark@tuxis.nl</div></div><div><br /></div>
<div x-em-replyforwardheader=""><br /></div>
<div>
<div>------ Original Message ------</div>
<div>From "Mark Schouten" <<a href="mailto:mark@tuxis.nl">mark@tuxis.nl</a>></div>
<div>To "Shannon Sterz" <<a href="mailto:s.sterz@proxmox.com">s.sterz@proxmox.com</a>></div>
<div>Cc "Proxmox Backup Server development discussion" <<a href="mailto:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>></div>
<div>Date 06/01/2025 20:07:43</div>
<div>Subject Re[6]: [pbs-devel] Authentication performance</div></div><div x-em-quote=""><br /></div>
<div id="x3ccf03dbbf4f4dc" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><blockquote cite="em808af812-c25f-4569-8e1c-231663ec413f@d1b4cc0c.com" type="cite" class="cite2">
<div><a href="https://bugzilla.proxmox.com/show_bug.cgi?id=6049">https://bugzilla.proxmox.com/show_bug.cgi?id=6049</a> has been created for this.</div><div><br /></div><div>Thanks!</div>
<div><br /></div><div id="signature_old" style="clear:both"><div style="margin: 0px; padding: 0px; box-sizing: content-box;">— </div><div style="margin: 0px; padding: 0px; box-sizing: content-box;">Mark Schouten</div><div style="margin: 0px; padding: 0px; box-sizing: content-box;">CTO, Tuxis B.V.</div><div style="margin: 0px; padding: 0px; box-sizing: content-box;">+31 318 200208 / mark@tuxis.nl</div></div><div><br /></div>
<div x-em-replyforwardheader=""><br /></div>
<div>
<div>------ Original Message ------</div>
<div>From "Shannon Sterz" <<a href="mailto:s.sterz@proxmox.com">s.sterz@proxmox.com</a>></div>
<div>To "Mark Schouten" <<a href="mailto:mark@tuxis.nl">mark@tuxis.nl</a>></div>
<div>Cc "Proxmox Backup Server development discussion" <<a href="mailto:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>></div>
<div>Date 20/12/2024 14:22:18</div>
<div>Subject Re: Re[4]: [pbs-devel] Authentication performance</div></div><div x-em-quote=""><br /></div>
<div id="xd7a8ca7f59be40f" class="plain"><blockquote cite="D6GK5TIN2LD4.1AXEF95IHZITS@proxmox.com" type="cite" class="cite2">
<div class="plain_line">On Thu Dec 19, 2024 at 10:56 AM CET, Mark Schouten wrote:</div>
<blockquote type="cite" class="cite">
<div class="plain_line"> Hi,</div>
<div class="plain_line"> </div>
<div class="plain_line"> We upgraded to 3.3 yesterday, not much gain to notice with regards to</div>
<div class="plain_line"> the new version or the change in keying. It’s still (obvioulsy) pretty</div>
<div class="plain_line"> busy.</div>
</blockquote>
<div class="plain_line"> </div>
<div class="plain_line">just be aware that the patch i linked to in my last mail has not been</div>
<div class="plain_line">packaged yet, so you wouldn't see the impact of that patch yet.</div>
<div class="plain_line"> </div>
<blockquote type="cite" class="cite2">
<div class="plain_line"> However, I also tried to remove some datastores, which failed with</div>
<div class="plain_line"> timeouts. PBS even stopped authenticating (so probably just working) all</div>
<div class="plain_line"> together for about 10 seconds, which was an unpleasant surprise.</div>
<div class="plain_line"> </div>
<div class="plain_line"> So looking into that further, I noticed the following logging:</div>
<div class="plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET</div>
<div class="plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client</div>
<div class="plain_line"> [::ffff]:42104] Unable to acquire lock</div>
<div class="plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error</div>
<div class="plain_line"> 4)</div>
<div class="plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET</div>
<div class="plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client</div>
<div class="plain_line"> [::ffff]:42144] Unable to acquire lock</div>
<div class="plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error</div>
<div class="plain_line"> 4)</div>
<div class="plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET</div>
<div class="plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client</div>
<div class="plain_line"> [::ffff]:47286] Unable to acquire lock</div>
<div class="plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error</div>
<div class="plain_line"> 4)</div>
<div class="plain_line"> Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET</div>
<div class="plain_line"> /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client</div>
<div class="plain_line"> [::ffff:]:45994] Unable to acquire lock</div>
<div class="plain_line"> "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error</div>
<div class="plain_line"> 4)</div>
<div class="plain_line"> </div>
<div class="plain_line"> Which surprised me, since this is a ’status’ call, which should not need</div>
<div class="plain_line"> locking of the datastore-config.</div>
<div class="plain_line"> </div>
<div class="plain_line"> <a href="https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=src/api2/admin/datastore.rs;h=c611f593624977defc49d6e4de2ab8185cfe09e9;hb=HEAD#l687" class="__cef_visited">https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=src/api2/admin/datastore.rs;h=c611f593624977defc49d6e4de2ab8185cfe09e9;hb=HEAD#l687</a></div>
<div class="plain_line"> does not lock the config, but</div>
<div class="plain_line"> </div>
<div class="plain_line"> <a href="https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=HEAD#l204" class="__cef_visited">https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=HEAD#l204</a></div>
<div class="plain_line"> does.</div>
<div class="plain_line"> </div>
<div class="plain_line"> So if I understand this correctly, every ’status’ call (30 per second in</div>
<div class="plain_line"> our case) locks the datastore-config exclusively. And also, every time</div>
<div class="plain_line"> ’status’ get called, the whole datastore-config gets loaded?</div>
</blockquote>
<div class="plain_line"> </div>
<div class="plain_line">probably, there are some comments about that there already, it might</div>
<div class="plain_line">make sense to open a bugzilla issue to discuss this further [1].</div>
<div class="plain_line"> </div>
<div class="plain_line">[1]: <a href="https://bugzilla.proxmox.com/">https://bugzilla.proxmox.com/</a></div>
<div class="plain_line"> </div>
<blockquote type="cite" class="cite2">
<div class="plain_line"> Is that something that could use some performance tuning?</div>
<div class="plain_line"> </div>
<div class="plain_line"> —</div>
<div class="plain_line"> Mark Schouten</div>
<div class="plain_line"> CTO, Tuxis B.V.</div>
<div class="plain_line"> +31 318 200208 / <a href="mailto:mark@tuxis.nl">mark@tuxis.nl</a></div>
<div class="plain_line"> </div>
<div class="plain_line"> </div>
<div class="plain_line"> ------ Original Message ------</div>
<div class="plain_line"> From "Shannon Sterz" <<a href="mailto:s.sterz@proxmox.com">s.sterz@proxmox.com</a>></div>
<div class="plain_line"> To "Mark Schouten" <<a href="mailto:mark@tuxis.nl">mark@tuxis.nl</a>></div>
<div class="plain_line"> Cc "Proxmox Backup Server development discussion"</div>
<div class="plain_line"> <<a href="mailto:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>></div>
<div class="plain_line"> Date 16/12/2024 12:51:47</div>
<div class="plain_line"> Subject Re: Re[2]: [pbs-devel] Authentication performance</div>
<div class="plain_line"> </div>
<div class="plain_line"> >On Mon Dec 16, 2024 at 12:23 PM CET, Mark Schouten wrote:</div>
<div class="plain_line"> >> Hi,</div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> ></div>
<div class="plain_line"> >> >would you mind sharing either `authkey.pub` or the output of the</div>
<div class="plain_line"> >> >following commands:</div>
<div class="plain_line"> >> ></div>
<div class="plain_line"> >> >head --lines=1 /etc/proxmox-backup/authkey.key</div>
<div class="plain_line"> >> >cat /etc/proxmox-backup/authkey.key | wc -l</div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> -----BEGIN RSA PRIVATE KEY-----</div>
<div class="plain_line"> >> 51</div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> So that is indeed the legacy method. We are going to upgrade our PBS’es</div>
<div class="plain_line"> >> on wednesday.</div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> ></div>
<div class="plain_line"> >> >The first should give the PEM header of the authkey whereas the second</div>
<div class="plain_line"> >> >provides the amount of lines that the key takes up in the file. Both</div>
<div class="plain_line"> >> >give an indication whether you are using the legacy RSA keys or newer</div>
<div class="plain_line"> >> >Ed25519 keys. The later should provide more performance, security should</div>
<div class="plain_line"> >> >not be affected much by this change. If the output of the commands look</div>
<div class="plain_line"> >> >like this:</div>
<div class="plain_line"> >> ></div>
<div class="plain_line"> >> >-----BEGIN PRIVATE KEY-----</div>
<div class="plain_line"> >> >3</div>
<div class="plain_line"> >> ></div>
<div class="plain_line"> >> >Then you are using the newer keys. There currently isn't a recommended</div>
<div class="plain_line"> >> >way to upgrade the keys. However, in theory you should be able to remove</div>
<div class="plain_line"> >> >the old keys, re-start PBS and it should just generate keys in the new</div>
<div class="plain_line"> >> >format. Note that this will logout anyone that is currently</div>
<div class="plain_line"> >> >authenticated and they'll have to re-authenticate.</div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> Seems like a good moment to update those keys as well.</div>
<div class="plain_line"> ></div>
<div class="plain_line"> >Sure, just be aware that you have to manually delete the key before</div>
<div class="plain_line"> >restarting the PBS. Upgrading alone won't affect the key. Ideally you'd</div>
<div class="plain_line"> >test this before rolling it out, if you can</div>
<div class="plain_line"> ></div>
<div class="plain_line"> >> >In general, tokens should still be fater to authenticate so we'd</div>
<div class="plain_line"> >> >recommend that you try to get your users to switch to token-based</div>
<div class="plain_line"> >> >authentication where possible. Improving performance there is a bit</div>
<div class="plain_line"> >> >trickier though, as it often comes with a security trade-off (in the</div>
<div class="plain_line"> >> >background we use yescrypt fo the authentication there, that</div>
<div class="plain_line"> >> >delibaretely adds a work factor). However, we may be able to improve</div>
<div class="plain_line"> >> >performance a bit via caching methods or similar.</div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> Yes, that might help. I’m also not sure if it actually is</div>
<div class="plain_line"> >> authentication, or if it is the datastore-call that the PVE-environments</div>
<div class="plain_line"> >> call. As you can see in your support issue 3153557, it looks like some</div>
<div class="plain_line"> >> requests loop through all datastores, before responding with a limited</div>
<div class="plain_line"> >> set of datastores.</div>
<div class="plain_line"> ></div>
<div class="plain_line"> >I looked at that ticket and yes, that is probably unrelated to</div>
<div class="plain_line"> >authentication.</div>
<div class="plain_line"> ></div>
<div class="plain_line"> >> For instance (and I’m a complete noob wrt Rust) but if I understand</div>
<div class="plain_line"> >><a href="https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=src/api2/admin/datastore.rs;h=11d2641b9ca2d2c92da1a85e4cb16d780368abd3;hb=HEAD#l1315">https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=src/api2/admin/datastore.rs;h=11d2641b9ca2d2c92da1a85e4cb16d780368abd3;hb=HEAD#l1315</a></div>
<div class="plain_line"> >> correcly, PBS loops through all the datastores, checks mount-status and</div>
<div class="plain_line"> >> config, and only starts filtering at line 1347. If I understand that</div>
<div class="plain_line"> >> correctly, in our case with over 1100 datastores, that might cause quite</div>
<div class="plain_line"> >> some load?</div>
<div class="plain_line"> ></div>
<div class="plain_line"> >Possible, yes, that would depend on your configuration. Are all of these</div>
<div class="plain_line"> >datastores defined with a backing device? Because if not, than this</div>
<div class="plain_line"> >should be fairly fast (as in, this should not actually touch the disks).</div>
<div class="plain_line"> >If they are, then yes this could be slow as each store would trigger at</div>
<div class="plain_line"> >least 2 stat calls afaict.</div>
<div class="plain_line"> ></div>
<div class="plain_line"> >In any case, it should be fine to move the `mount_status` check after</div>
<div class="plain_line"> >the `if allowed || allow_id` check from what i can tell. Not sure why</div>
<div class="plain_line"> >we'd need to check the mount_status for a datastore we won't include in</div>
<div class="plain_line"> >the resulsts anyway. Same goes for parsing the store config imo. Send a</div>
<div class="plain_line"> >patch for that [1].</div>
<div class="plain_line"> ></div>
<div class="plain_line"> >[1]: <a href="https://lore.proxmox.com/pbs-devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u">https://lore.proxmox.com/pbs-devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u</a></div>
<div class="plain_line"> ></div>
<div class="plain_line"> >></div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> Thanks,</div>
<div class="plain_line"> >></div>
<div class="plain_line"> >> —</div>
<div class="plain_line"> >> Mark Schouten</div>
<div class="plain_line"> >> CTO, Tuxis B.V.</div>
<div class="plain_line"> >> +31 318 200208 / <a href="mailto:mark@tuxis.nl">mark@tuxis.nl</a></div>
<div class="plain_line"> ></div>
<div class="plain_line"> ></div>
</blockquote>
<div class="plain_line"> </div>
<div class="plain_line"> </div>
</blockquote></div>
</blockquote></div>
</body></html>