[pbs-devel] Too many open files

[pbs-devel] Too many open files

[Mark Schouten]

Hi,

On our shared PBS-server, on busy moments, we get the following error:
Sep  7 00:00:35 pbs001 proxmox-backup-proxy[9288]: error accepting tcp 
connection: Too many open files (os error 24)

This is a box with quite a lot of datastores, but looking at the open 
files setting:
root@pbs001:/var/log# cat /proc/sys/fs/file-max
9223372036854775807

That should be enough. :)

Does PBS itself limit the open files somewhere?

Thanks,


—
Mark Schouten, CTO
Tuxis B.V.
mark@tuxis.nl

Re: [pbs-devel] Too many open files

[Thomas Lamprecht]

Hi,

Am 07/09/2022 um 14:02 schrieb Mark Schouten:
> On our shared PBS-server, on busy moments, we get the following error:
> Sep  7 00:00:35 pbs001 proxmox-backup-proxy[9288]: error accepting tcp connection: Too many open files (os error 24)
> 

How many FDs are open for the proxy?

ls -1 /proc/$(pidof proxmox-backup-proxy)/fd | wc -l

> This is a box with quite a lot of datastores, but looking at the open files setting:
> root@pbs001:/var/log# cat /proc/sys/fs/file-max
> 9223372036854775807

This is not the relevant limit though, this is the upper limit of the
kernel for file handles
https://sysctl-explorer.net/fs/filemax%20&%20filenr/

The per-process file (hard) limit would be:
https://sysctl-explorer.net/fs/nr_open/

And as that mentions, that again depends on the rlimit, check the default (soft)
imits with

- as root:
  ulimit -a

- for the backup user
  su - backup -s /bin/bash -c "ulimit -n $[1024*1024]"


Or probably even easier, check the /proc entry of the proxy and api daemons:

cat /proc/$(pidof proxmox-backup-proxy)/limits
cat /proc/$(pidof proxmox-backup-api)/limits

> 
> That should be enough. 😄
> 
> Does PBS itself limit the open files somewhere?

No, but it doesn't increase it's soft limit to the hard limit, something we
want to do in the future.

As workaround try increasing soft (and possible hard) limit for root and backup
user, you can do that by editing `/etc/security/limits.conf`, adding lines like:

backup soft nofile 524288
root soft nofile 524288

(should set soft to current hard limit)

Or alternatively use a systemd service override to make it really process specific:

# systemctl edit proxmox-backup-proxy.service

There add a "[Service]" Section, if not already existing and a "LimitNOFILE=X" below,
e.g.:
 
[Service]
LimitNOFILE=524288

Save and close the editor and restart the service, then do the same for the
proxmox-backup.service

As said, we plan to upper that soft limit too in the future.

cheers,
Thomas

Re: [pbs-devel] Too many open files

[Mark Schouten]

Hi,



>How many FDs are open for the proxy?
>
>ls -1 /proc/$(pidof proxmox-backup-proxy)/fd | wc -l


This was not that much at this moment, but aroung 0.00 hours, the 
counter increases, I suspect. ;)

>No, but it doesn't increase it's soft limit to the hard limit, something we
>want to do in the future.
>
>As workaround try increasing soft (and possible hard) limit for root and backup
>user, you can do that by editing `/etc/security/limits.conf`, adding lines like:
>
>backup soft nofile 524288
>root soft nofile 524288
>
>(should set soft to current hard limit)

This did not work for me.

>Or alternatively use a systemd service override to make it really process specific:
>
># systemctl edit proxmox-backup-proxy.service
>
>There add a "[Service]" Section, if not already existing and a "LimitNOFILE=X" below,
>e.g.:
>
>[Service]
>LimitNOFILE=524288
>
>Save and close the editor and restart the service, then do the same for the
>proxmox-backup.service
>
>As said, we plan to upper that soft limit too in the future.

This worked like a charm. Let’s see if it helps tonight during peak 
hours.

Thanks!


—
Mark Schouten, CTO
Tuxis B.V.
mark@tuxis.nl