[PVE-User] Peak load at 7.30AM...

[PVE-User] Peak load at 7.30AM...

[Marco Gaiarin]

Situation: a debian stretch mostly 'samba server' for a 150+ clients, in a
couple of phisical server; VM get replicated between the two server every 30
minutes.

Very frequently at 7.30 the VM got a high peak rate, becaming mostly
irresponsive; after fiddling a bit, i've added a watchdog LOAD limit, and
the VM now reboot.

Looking at logs, it seems caused by the replica of the 7.30:

	Apr 26 07:30:12 vdmsv1 qemu-ga: info: guest-ping called
	Apr 26 07:30:13 vdmsv1 qemu-ga: info: guest-fsfreeze called

and after some (6 to 8 minutes) watchdog reset it:

	Apr 26 07:36:11 vdmsv1 watchdog[2525]: loadavg 57 33 14 is higher than the given threshold 80 32 16!
	Apr 26 07:36:11 vdmsv1 watchdog[2525]: shutting down the system because of error 253 = 'load average too high'


Some notes:

1) the replica run every 30 minutes; no other of the 47 replicas of the day
  seems sufficient to trigger a reboot.

2) the phisical server during the high peak seems totally unaffected (no
  high iodelay, no sensible load/cpu...).

3) at 7.30 there's no user (they arrive around 8.15).


I'm doing some hypotesis; for example debian by default rotate logs at 6.30,
but looking at file dates logs are completely rotated before the 7.00, so
the 7.00 replica could have triggered the reboot...


Someone have some hint on how can i debug this!?


Thanks.

-- 
  E allora osservi gli altri giocare
  e` un gioco strano devi imparare			(E. Bennato)

Re: [PVE-User] Peak load at 7.30AM...

[Laurent CARON]

Hi,

What about smbstatus ? No clients accessing shares ? No automated AV 
scan, ... ?


Le 26/04/2023 à 12:48, Marco Gaiarin a écrit :
> Situation: a debian stretch mostly 'samba server' for a 150+ clients, in a
> couple of phisical server; VM get replicated between the two server every 30
> minutes.
>
> Very frequently at 7.30 the VM got a high peak rate, becaming mostly
> irresponsive; after fiddling a bit, i've added a watchdog LOAD limit, and
> the VM now reboot.
>
> Looking at logs, it seems caused by the replica of the 7.30:
>
> 	Apr 26 07:30:12 vdmsv1 qemu-ga: info: guest-ping called
> 	Apr 26 07:30:13 vdmsv1 qemu-ga: info: guest-fsfreeze called
>
> and after some (6 to 8 minutes) watchdog reset it:
>
> 	Apr 26 07:36:11 vdmsv1 watchdog[2525]: loadavg 57 33 14 is higher than the given threshold 80 32 16!
> 	Apr 26 07:36:11 vdmsv1 watchdog[2525]: shutting down the system because of error 253 = 'load average too high'
>
>
> Some notes:
>
> 1) the replica run every 30 minutes; no other of the 47 replicas of the day
>    seems sufficient to trigger a reboot.
>
> 2) the phisical server during the high peak seems totally unaffected (no
>    high iodelay, no sensible load/cpu...).
>
> 3) at 7.30 there's no user (they arrive around 8.15).
>
>
> I'm doing some hypotesis; for example debian by default rotate logs at 6.30,
> but looking at file dates logs are completely rotated before the 7.00, so
> the 7.00 replica could have triggered the reboot...
>
>
> Someone have some hint on how can i debug this!?
>
>
> Thanks.
>

Re: [PVE-User] Peak load at 7.30AM...

[Marco Gaiarin]

> I'm doing some hypotesis; for example debian by default rotate logs at 6.30,
> but looking at file dates logs are completely rotated before the 7.00, so
> the 7.00 replica could have triggered the reboot...

OK, found.

It was effectively the daily tasks; not logrotate, but a task/script that do
a set of 'find' in a virtual disk hosted on a slow ZFS pool (see my other
post).

Seems that doing a 'findstorm' on that volume can lead load to sky...


For now, i've added a 'sleep 1' between find calls; i don't need the script
end in precise time.

-- 
  La condivisione è il segreto di tutto...		(Roberto Colonello)