[pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[dea]

Hello everyone,

I've been using PBS on NFS storage (on several installations) for some 
time now, and I'm used to some kind of performance.
For example, a full SSD datastore (on the NFS datastore) I know takes 2 
hours for a full garbage collect, etc. (datastore from 25 to 300 Tbytes).
This was always the case until release 2.3-2.
After the upgrade to 2.3-3 something changed (ok also to kernel 5.15-85) 
the same datastore that used to be completed in 2 hours, now after 3 
days is 95% in marked mode.

Something does not add up for me (datastore side has not been touched a 
comma), ditto for PBS configuration.

The PBS is in production, as soon as I can I would like to upgrade it to 
kernel 6.1 .... I don't know what could be the cause.

I don't understand.....

Many thanks
Luca

Re: [pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[dea]

Hi can confirm !

Downgrade via dpkg from 2.3-3 to 2.3-2 (and change kernel to 6.1) and 
works very fast, as usually.

I don't know if the problem is in the kernel or in the 2.3-3 package, 
but this way it works as it should.

The system is in production, so I can't do too many tests and reboots...

I was exasperated by the slowness, so I made two changes at once (I know 
that diagnostically it's the worst solution, but not having time or a 
way to give too much disruption I couldn't do one test at a time).

Luca

Re: [pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[Thomas Lamprecht]

Hi,

Am 09/03/2023 um 13:38 schrieb dea:
> Downgrade via dpkg from 2.3-3 to 2.3-2 (and change kernel to 6.1) and works very fast, as usually.
> 

what packages where in the "bad" update set? Can you please check /var/log/apt/history.log ?

> I don't know if the problem is in the kernel or in the 2.3-3 package, but this way it works as it should.
> 

Hmm, we moved proxmox-backup-server version 2.3.3-1 to no-subscription over a month
ago (2023-02-10) and had not seen any wide-spreading reports of general problems
introduced by that version.

So if a new kernel was pulled in too it could be indeed related to that.
The used Hardware (CPU, NIC, ...) would be good to know too, maybe it's a regression
specific to some component of your system.


> The system is in production, so I can't do too many tests and reboots...
> 
> I was exasperated by the slowness, so I made two changes at once (I know that diagnostically it's the worst solution, but not having time or a way to give too much disruption I couldn't do one test at a time).

It'd be really great if you could find some time to test 2.3.3-1 again while keeping
the newer 6.1 kernel booted. Downgrading that back again shouldn't require that much
time (at least less than switching kernel), and would help to tell if PBS itself can
be excluded from the regression hunt.

 - thomas

Re: [pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[dea]

Hi Thomas,

at the moment the PBS is using release 2.3-2 with kernel 6.1.10-1 and 
everything is working properly, with the expected performance.
I too think the problem is not with PBS 2.3-3 but with the kernel, and 
on my system kernel 6.1.10 runs really well.

The problem I have identified with kernel 5.15.85 is incredibly slow 
performance on NFS storage.

3 days to finish a garbage collector on a 25 Tbyte SSD datastore (in 
NFS) versus 3 hours now.

Yes, I think the best analysis is to upgrade to PBS 2.3-3 while keeping 
the system running kernel 6.1.10.

I will let you know

Thank you very much

Luca

Re: [pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[dea]

Hi Thomas,

what I am about to say is not related to the problem, but it has a sense 
that connects it.

If it were possible during the garbage collect function to introduce 
"checkpoints" so that in the case of a reboot or upgrade a days' worth 
of work is not thrown away, it would really be a great step forward.
Now I use about 300 Tbytes of hybrid storage (HDD with acceleration SSD) 
and about 25 Tbytes of full SSD and the garbage collect function is 
really onerous on the hybrid storage.
If I were to increase the capacity to 1 Pbyte and more, it would be 
really difficult to manage.


Thanks

Luca

Re: [pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[dea]

Hi Thomas,

I can confirm a kernel problem in my server.

Now, on kernel 6.2.6 (Enterprise repository) PBS is much faster (running 
on 2.4-1)

Thanks

Luca

Re: [pbs-devel] Possible problem on NFS storage with release 2-3-3 (??)

[dea]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

Hi Thomas,

my main "resistance" to trying to upgrade or version rollbacks is not 
the operation itself....
But this:


My garbage collect on hybrid, non-full SSD (NFS) datastores takes at 
least 10 days (that's over 300 Tbytes of storage).

If there was a system to keep track of the garbage collect parsing point 
before a reboot or upgrade would be a great thing, so that it could then 
be resumed from the breakpoint.

If I update the PBS the garbage collects are closed and restarted from 
scratch, ditto for a reboot.

Luca

[-- Attachment #2.1: Type: text/html, Size: 873 bytes --]

[-- Attachment #2.2: 0xcqlUg3vgVffYCE.png --]
[-- Type: image/png, Size: 19803 bytes --]