Two nodes asymmetrical cluster, backup, stall of one node.

[Marco Gaiarin <gaio@lilliput.linux.it>]

I've hit a trouble i cannot understand very well; we have some asymmetrical
clusters, composed of one main server and a 'backup', a little underpowered, server
that act as backup, exporting their fs via NFS.

At every backup run (weekly), the backup node got some pressure (this is
expected and common to other clusters), kernel start to complain as:

 2026-08-30T00:18:43.125643+02:00 pdpve2 pvescheduler[1474088]: jobs: cfs-lock 'file-jobs_cfg' error: got lock request timeout
 2026-08-30T00:18:43.723075+02:00 pdpve2 pvescheduler[1474087]: replication: cfs-lock 'file-replication_cfg' error: got lock request timeout
 [...]
 2026-08-30T00:49:59.269199+02:00 pdpve2 kernel: [481202.580590] INFO: task pvescheduler:1478618 blocked for more than 122 seconds.

after that, backup node get 'offline' (from main node), pveproxy respond in
backup node but i canot login (local PAM account or domain ones).

If i SSH on the backup node, all seems normal and working, no service
stopped, a little test VM running on it working as expected.

Only a note, there what seems a stalled backup job:

 root@pdpve2:~# ps aux | grep pves[r]
 root     1497581  0.0  0.6 308788 110928 ?       Ds   Aug30   0:00 /usr/bin/perl -T /usr/bin/pvesr prepare-local-job 100-0 --scan local-zfs local-zfs:vm-100-disk-0 --last_sync 1788035400

i've trid to kill it, even with '-9', but whitout success.


If i reboot the node, all come back as normal.


Someone have some clue?


If i have a set ov VM on node A and a set of VM on node B, and i run a
backup of all VMs, there's some way to 'serialize' it, eg, do backup of VMs
of node A and after that backup of VM on node B?

I suppose that part of the trouble came fron the fact that backup start in
parrallels on both nodes, but node B is also the destination of backup, so
it get 3 time the IO (write from node A, read from node B, write to node B).

I can surely setup TWO backup job, but... thanks.

--