From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pbs-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 14CD21FF163 for <inbox@lore.proxmox.com>; Thu, 19 Dec 2024 10:57:10 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 934142FB70; Thu, 19 Dec 2024 10:57:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=tuxis.nl; s=mail; h=from:reply-to:subject:date:message-id:to:cc:mime-version:content-type: in-reply-to:references; bh=ITi6VX6jN8mza89z4hxqLRkIqzvTe7y0U9ialr0LfJg=; b=VpsPR+QB1K2FsCBdD1Ki0kEHM/oEzi+17BVsmzFY6YRj7hNbHHbkwmS+wAQt0FyNqrCaM0aJTFCPN EOVfOCZ86vgwB+8aXiTCd8T8vNRTl4CPJZUtmpBfuGicwgvMtnSUlokF64vG0Kp4HEY7bJH0FF6GG6 R8rL+0Q0hBrASrsY6qRgu83EQejlvCilTkfWNcYQ/VV2Khn1lpsyPDQPnhoNc1s62nt4P/oWxJ07+P bpO6D7VrqxW5TjF+B+EzJ3XLtg1gkOjnmfgtIzp0KmjKmNL226yD7Is3TfG5jmMSIgvFoDhOQI6qNh CTa/rafSd2ZcT+T02iHw33BAxrhV/rA== X-Footer: dHV4aXMubmw= From: "Mark Schouten" <mark@tuxis.nl> To: "Shannon Sterz" <s.sterz@proxmox.com> Date: Thu, 19 Dec 2024 09:56:23 +0000 Message-Id: <em2d4327d4-4607-458e-880d-9b6bfba58f0a@ef757d37.com> In-Reply-To: <D6D3QC6Y5H4S.1QHYHPHXK6RVR@proxmox.com> References: <ema6032dba-8585-4377-bec1-11a37159087c@192dede7.com> <D6D024F1JMI5.QDFXDKCQMUCJ@proxmox.com> <embeb48874-d400-4e69-ae0f-2cc56a39d592@93f95f61.com> <D6D3QC6Y5H4S.1QHYHPHXK6RVR@proxmox.com> User-Agent: eM_Client/10.1.4828.0 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.111 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy HTML_MESSAGE 0.001 HTML included in message KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] Authentication performance X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Mark Schouten <mark@tuxis.nl>, Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Cc: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Content-Type: multipart/mixed; boundary="===============2087944167108576542==" Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com> --===============2087944167108576542== Content-Type: multipart/alternative; boundary="------=_MB028D2454-D79A-484D-AE8C-58DA2711EBCE" --------=_MB028D2454-D79A-484D-AE8C-58DA2711EBCE Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi, We upgraded to 3.3 yesterday, not much gain to notice with regards to=20 the new version or the change in keying. It=E2=80=99s still (obvioulsy) pre= tty=20 busy. However, I also tried to remove some datastores, which failed with=20 timeouts. PBS even stopped authenticating (so probably just working) all=20 together for about 10 seconds, which was an unpleasant surprise. So looking into that further, I noticed the following logging: Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET=20 /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client=20 [::ffff]:42104] Unable to acquire lock=20 "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error=20 4) Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET=20 /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client=20 [::ffff]:42144] Unable to acquire lock=20 "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error=20 4) Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET=20 /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client=20 [::ffff]:47286] Unable to acquire lock=20 "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error=20 4) Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET=20 /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client=20 [::ffff:]:45994] Unable to acquire lock=20 "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os error=20 4) Which surprised me, since this is a =E2=80=99status=E2=80=99 call, which sh= ould not need=20 locking of the datastore-config. https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin= /datastore.rs;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l687= =20 does not lock the config, but https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-datastore/= src/datastore.rs;h=3D0801b4bf6b25eaa6f306c7b39ae2cfe81b4782e1;hb=3DHEAD#l20= 4=20 does. So if I understand this correctly, every =E2=80=99status=E2=80=99 call (30= per second in=20 our case) locks the datastore-config exclusively. And also, every time=20 =E2=80=99status=E2=80=99 get called, the whole datastore-config gets loaded= ? Is that something that could use some performance tuning? =E2=80=94 Mark Schouten CTO, Tuxis B.V. +31 318 200208 / mark@tuxis.nl ------ Original Message ------ >From "Shannon Sterz" <s.sterz@proxmox.com> To "Mark Schouten" <mark@tuxis.nl> Cc "Proxmox Backup Server development discussion"=20 <pbs-devel@lists.proxmox.com> Date 16/12/2024 12:51:47 Subject Re: Re[2]: [pbs-devel] Authentication performance >On Mon Dec 16, 2024 at 12:23 PM CET, Mark Schouten wrote: >> Hi, >> >> > >> >would you mind sharing either `authkey.pub` or the output of the >> >following commands: >> > >> >head --lines=3D1 /etc/proxmox-backup/authkey.key >> >cat /etc/proxmox-backup/authkey.key | wc -l >> >> -----BEGIN RSA PRIVATE KEY----- >> 51 >> >> So that is indeed the legacy method. We are going to upgrade our PBS=E2= =80=99es >> on wednesday. >> >> > >> >The first should give the PEM header of the authkey whereas the second >> >provides the amount of lines that the key takes up in the file. Both >> >give an indication whether you are using the legacy RSA keys or newer >> >Ed25519 keys. The later should provide more performance, security shou= ld >> >not be affected much by this change. If the output of the commands loo= k >> >like this: >> > >> >-----BEGIN PRIVATE KEY----- >> >3 >> > >> >Then you are using the newer keys. There currently isn't a recommended >> >way to upgrade the keys. However, in theory you should be able to remo= ve >> >the old keys, re-start PBS and it should just generate keys in the new >> >format. Note that this will logout anyone that is currently >> >authenticated and they'll have to re-authenticate. >> >> Seems like a good moment to update those keys as well. > >Sure, just be aware that you have to manually delete the key before >restarting the PBS. Upgrading alone won't affect the key. Ideally you'd >test this before rolling it out, if you can > >> >In general, tokens should still be fater to authenticate so we'd >> >recommend that you try to get your users to switch to token-based >> >authentication where possible. Improving performance there is a bit >> >trickier though, as it often comes with a security trade-off (in the >> >background we use yescrypt fo the authentication there, that >> >delibaretely adds a work factor). However, we may be able to improve >> >performance a bit via caching methods or similar. >> >> Yes, that might help. I=E2=80=99m also not sure if it actually is >> authentication, or if it is the datastore-call that the PVE-environment= s >> call. As you can see in your support issue 3153557, it looks like some >> requests loop through all datastores, before responding with a limited >> set of datastores. > >I looked at that ticket and yes, that is probably unrelated to >authentication. > >> For instance (and I=E2=80=99m a complete noob wrt Rust) but if I unders= tand >>https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/adm= in/datastore.rs;h=3D11d2641b9ca2d2c92da1a85e4cb16d780368abd3;hb=3DHEAD#l131= 5 >> correcly, PBS loops through all the datastores, checks mount-status and >> config, and only starts filtering at line 1347. If I understand that >> correctly, in our case with over 1100 datastores, that might cause quit= e >> some load? > >Possible, yes, that would depend on your configuration. Are all of these >datastores defined with a backing device? Because if not, than this >should be fairly fast (as in, this should not actually touch the disks). >If they are, then yes this could be slow as each store would trigger at >least 2 stat calls afaict. > >In any case, it should be fine to move the `mount_status` check after >the `if allowed || allow_id` check from what i can tell. Not sure why >we'd need to check the mount_status for a datastore we won't include in >the resulsts anyway. Same goes for parsing the store config imo. Send a >patch for that [1]. > >[1]: https://lore.proxmox.com/pbs-devel/20241216115044.208595-1-s.sterz@pr= oxmox.com/T/#u > >> >> >> Thanks, >> >> =E2=80=94 >> Mark Schouten >> CTO, Tuxis B.V. >> +31 318 200208 / mark@tuxis.nl > > --------=_MB028D2454-D79A-484D-AE8C-58DA2711EBCE Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head> <style id=3D"css_styles" type=3D"text/css"><!--blockquote.cite { margin-lef= t: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-le= ft: 1px solid #cccccc } blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px;= padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding= -top: 0px; } a img { border: 0px; } table { border-collapse: collapse; } li[style=3D'text-align: center;'], li[style=3D'text-align: center; '], li[s= tyle=3D'text-align: right;'], li[style=3D'text-align: right; '] { list-sty= le-position: inside;} body { font-family: Helvetica; font-size: 9pt; } .quote { margin-left: 1em; margin-right: 1em; border-left: 5px #ebebeb soli= d; padding-left: 0.3em; } a.em-mention[href] { text-decoration: none; color: inherit; border-radius:= 3px; padding-left: 2px; padding-right: 2px; background-color: #e2e2e2; } --></style></head> <body style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; line-br= eak: after-white-space;"><div>Hi,</div><div><br /></div><div>We upgraded to = 3.3 yesterday, not much gain to notice with regards to the new version or= the change in keying. It=E2=80=99s still (obvioulsy) pretty busy.</div><div= ><br /></div><div>However, I also tried to remove some datastores, which fa= iled with timeouts. PBS even stopped authenticating (so probably just worki= ng) all together for about 10 seconds, which was an unpleasant surprise.</d= iv><div><br /></div><div>So looking into that further, I noticed the follow= ing logging:</div><div>Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]:= GET /api2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client [::f= fff]:42104] Unable to acquire lock "/etc/proxmox-backup/.datastore.lck" - I= nterrupted system call (os error 4)</div><div>Dec 18 16:14:32 pbs005 proxmo= x-backup-proxy[39143]: GET /api2/json/admin/datastore/XXXXXX/status: 400 Ba= d Request: [client [::ffff]:42144] Unable to acquire lock "/etc/proxmox-bac= kup/.datastore.lck" - Interrupted system call (os error 4)</div><div>Dec 18 = 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET /api2/json/admin/datastor= e/XXXXXX/status: 400 Bad Request: [client [::ffff]:47286] Unable to acquire = lock "/etc/proxmox-backup/.datastore.lck" - Interrupted system call (os er= ror 4)</div><div>Dec 18 16:14:32 pbs005 proxmox-backup-proxy[39143]: GET /a= pi2/json/admin/datastore/XXXXXX/status: 400 Bad Request: [client [::ffff:]:= 45994] Unable to acquire lock "/etc/proxmox-backup/.datastore.lck" - Interr= upted system call (os error 4)</div><div><br /></div><div>Which surprised m= e, since this is a =E2=80=99status=E2=80=99 call, which should not need loc= king of the datastore-config.</div><div><br /></div><div><a href=3D"https:/= /git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datast= ore.rs;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l687">https:/= /git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dsrc/api2/admin/datast= ore.rs;h=3Dc611f593624977defc49d6e4de2ab8185cfe09e9;hb=3DHEAD#l687</a>=C2= =A0does not lock the config, but=C2=A0</div> <div><br /></div><div><a href=3D"https://git.proxmox.com/?p=3Dproxmox-backu= p.git;a=3Dblob;f=3Dpbs-datastore/src/datastore.rs;h=3D0801b4bf6b25eaa6f306c= 7b39ae2cfe81b4782e1;hb=3DHEAD#l204">https://git.proxmox.com/?p=3Dproxmox-ba= ckup.git;a=3Dblob;f=3Dpbs-datastore/src/datastore.rs;h=3D0801b4bf6b25eaa6f3= 06c7b39ae2cfe81b4782e1;hb=3DHEAD#l204</a>=C2=A0does.</div><div><br /></div>= <div>So if I understand this correctly, every =E2=80=99status=E2=80=99 call = (30 per second in our case) locks the datastore-config exclusively. And al= so, every time =E2=80=99status=E2=80=99 get called, the whole datastore-con= fig gets loaded?</div><div><br /></div><div>Is that something that could us= e some performance tuning?</div><div><br /></div><div id=3D"signature_old"= style=3D"clear:both"><div style=3D"margin: 0px; padding: 0px; box-sizing: c= ontent-box;">=E2=80=94=C2=A0</div><div style=3D"margin: 0px; padding: 0px;= box-sizing: content-box;">Mark Schouten</div><div style=3D"margin: 0px; pad= ding: 0px; box-sizing: content-box;">CTO, Tuxis B.V.</div><div style=3D"mar= gin: 0px; padding: 0px; box-sizing: content-box;">+31 318 200208 /=C2=A0mar= k@tuxis.nl</div></div><div><br /></div> <div x-em-replyforwardheader=3D""><br /></div> <div> <div>------ Original Message ------</div> <div>From "Shannon Sterz" <<a href=3D"mailto:s.sterz@proxmox.com">s.ster= z@proxmox.com</a>></div> <div>To "Mark Schouten" <<a href=3D"mailto:mark@tuxis.nl">mark@tuxis.nl<= /a>></div> <div>Cc "Proxmox Backup Server development discussion" <<a href=3D"mailt= o:pbs-devel@lists.proxmox.com">pbs-devel@lists.proxmox.com</a>></div> <div>Date 16/12/2024 12:51:47</div> <div>Subject Re: Re[2]: [pbs-devel] Authentication performance</div></div><= div x-em-quote=3D""><br /></div> <div id=3D"x6df669de829a4a9" class=3D"plain"><blockquote cite=3D"D6D3QC6Y5H= 4S.1QHYHPHXK6RVR@proxmox.com" type=3D"cite" class=3D"cite2"> <div class=3D"plain_line">On Mon Dec 16, 2024 at 12:23 PM CET, Mark Schoute= n wrote:</div> <blockquote type=3D"cite" class=3D"cite"> <div class=3D"plain_line"> Hi,</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >would you mind sharing either `authkey.pub`= or the output of the</div> <div class=3D"plain_line"> >following commands:</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >head --lines=3D1 /etc/proxmox-backup/authkey= .key</div> <div class=3D"plain_line"> >cat /etc/proxmox-backup/authkey.key | wc -l<= /div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> -----BEGIN RSA PRIVATE KEY-----</div> <div class=3D"plain_line"> 51</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> So that is indeed the legacy method. We are goin= g to upgrade our PBS=E2=80=99es</div> <div class=3D"plain_line"> on wednesday.</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >The first should give the PEM header of the= authkey whereas the second</div> <div class=3D"plain_line"> >provides the amount of lines that the key ta= kes up in the file. Both</div> <div class=3D"plain_line"> >give an indication whether you are using the = legacy RSA keys or newer</div> <div class=3D"plain_line"> >Ed25519 keys. The later should provide more= performance, security should</div> <div class=3D"plain_line"> >not be affected much by this change. If the= output of the commands look</div> <div class=3D"plain_line"> >like this:</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >-----BEGIN PRIVATE KEY-----</div> <div class=3D"plain_line"> >3</div> <div class=3D"plain_line"> ></div> <div class=3D"plain_line"> >Then you are using the newer keys. There cur= rently isn't a recommended</div> <div class=3D"plain_line"> >way to upgrade the keys. However, in theory= you should be able to remove</div> <div class=3D"plain_line"> >the old keys, re-start PBS and it should jus= t generate keys in the new</div> <div class=3D"plain_line"> >format. Note that this will logout anyone th= at is currently</div> <div class=3D"plain_line"> >authenticated and they'll have to re-authent= icate.</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> Seems like a good moment to update those keys= as well.</div> </blockquote> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">Sure, just be aware that you have to manually del= ete the key before</div> <div class=3D"plain_line">restarting the PBS. Upgrading alone won't affect= the key. Ideally you'd</div> <div class=3D"plain_line">test this before rolling it out, if you can</div> <div class=3D"plain_line">=C2=A0</div> <blockquote type=3D"cite" class=3D"cite2"> <div class=3D"plain_line"> >In general, tokens should still be fater to= authenticate so we'd</div> <div class=3D"plain_line"> >recommend that you try to get your users to= switch to token-based</div> <div class=3D"plain_line"> >authentication where possible. Improving per= formance there is a bit</div> <div class=3D"plain_line"> >trickier though, as it often comes with a se= curity trade-off (in the</div> <div class=3D"plain_line"> >background we use yescrypt fo the authentica= tion there, that</div> <div class=3D"plain_line"> >delibaretely adds a work factor). However, w= e may be able to improve</div> <div class=3D"plain_line"> >performance a bit via caching methods or sim= ilar.</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> Yes, that might help. I=E2=80=99m also not sure= if it actually is</div> <div class=3D"plain_line"> authentication, or if it is the datastore-call t= hat the PVE-environments</div> <div class=3D"plain_line"> call. As you can see in your support issue 31535= 57, it looks like some</div> <div class=3D"plain_line"> requests loop through all datastores, before res= ponding with a limited</div> <div class=3D"plain_line"> set of datastores.</div> </blockquote> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">I looked at that ticket and yes, that is probably = unrelated to</div> <div class=3D"plain_line">authentication.</div> <div class=3D"plain_line">=C2=A0</div> <blockquote type=3D"cite" class=3D"cite2"> <div class=3D"plain_line"> For instance (and I=E2=80=99m a complete noob wr= t Rust) but if I understand</div> <div class=3D"plain_line"> <a href=3D"https://git.proxmox.com/?p=3Dproxmox-= backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3D11d2641b9ca2d2c92da= 1a85e4cb16d780368abd3;hb=3DHEAD#l1315">https://git.proxmox.com/?p=3Dproxmox= -backup.git;a=3Dblob;f=3Dsrc/api2/admin/datastore.rs;h=3D11d2641b9ca2d2c92d= a1a85e4cb16d780368abd3;hb=3DHEAD#l1315</a></div> <div class=3D"plain_line"> correcly, PBS loops through all the datastores,= checks mount-status and</div> <div class=3D"plain_line"> config, and only starts filtering at line 1347.= If I understand that</div> <div class=3D"plain_line"> correctly, in our case with over 1100 datastores= , that might cause quite</div> <div class=3D"plain_line"> some load?</div> </blockquote> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">Possible, yes, that would depend on your configur= ation. Are all of these</div> <div class=3D"plain_line">datastores defined with a backing device? Because = if not, than this</div> <div class=3D"plain_line">should be fairly fast (as in, this should not act= ually touch the disks).</div> <div class=3D"plain_line">If they are, then yes this could be slow as each= store would trigger at</div> <div class=3D"plain_line">least 2 stat calls afaict.</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">In any case, it should be fine to move the `mount= _status` check after</div> <div class=3D"plain_line">the `if allowed || allow_id` check from what i ca= n tell. Not sure why</div> <div class=3D"plain_line">we'd need to check the mount_status for a datasto= re we won't include in</div> <div class=3D"plain_line">the resulsts anyway. Same goes for parsing the st= ore config imo. Send a</div> <div class=3D"plain_line">patch for that [1].</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">[1]: <a href=3D"https://lore.proxmox.com/pbs-deve= l/20241216115044.208595-1-s.sterz@proxmox.com/T/#u">https://lore.proxmox.co= m/pbs-devel/20241216115044.208595-1-s.sterz@proxmox.com/T/#u</a></div> <div class=3D"plain_line">=C2=A0</div> <blockquote type=3D"cite" class=3D"cite2"> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> Thanks,</div> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line"> =E2=80=94</div> <div class=3D"plain_line"> Mark Schouten</div> <div class=3D"plain_line"> CTO, Tuxis B.V.</div> <div class=3D"plain_line"> +31 318 200208 / <a href=3D"mailto:mark@tuxis.nl= ">mark@tuxis.nl</a></div> </blockquote> <div class=3D"plain_line">=C2=A0</div> <div class=3D"plain_line">=C2=A0</div> </blockquote></div> </body></html> --------=_MB028D2454-D79A-484D-AE8C-58DA2711EBCE-- --===============2087944167108576542== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel --===============2087944167108576542==--