From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 01C121FF0AE for ; Tue, 01 Sep 2026 12:55:55 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7BB56215D5; Tue, 01 Sep 2026 12:55:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=antreich.com; h= cc:content-transfer-encoding:content-type:content-type:date:from :from:in-reply-to:message-id:mime-version:references:reply-to :subject:subject:to:to; s=2025; bh=amiI8Qo5ex5TBqVTKRQJrwyVCfCqY QraYsxYjh0jW5w=; b=mdCdzXZhD2Az8I0LEYQxQyluaqBPmYuoF0DykLHgR/PjI MfDtQQKDclQ0cxGrSQzocT2JnVMuEpkdTRIwaKFIYkdcIGB+unIy1Gm9fcYA1FQX 7aJc3IV8/5vIDQ6HwJ31XjJvLFhre58+IyC4Dxx4JgQU6+WTofXYg6k+h5NMj+oi z8MNzFh1UvEUVgzWjtcACayev0VOYbowIM6so6CqUwRSY11acm7X1OJwxQ1RV43m GUv09074FyoBM8SOE3A1lYTopA454QF8+DK09MGSAsUVONfmJPoRV1oBCHUqHzDZ kch7wydqQFAAMaaunvzPn3qOBpkpFCD0gj7tfcSLg== Date: Tue, 01 Sep 2026 10:55:22 +0000 Message-ID: <20260901105522.Horde.vgrn9fzQ416tczxJi0dVq0Y@drive.antreich.com> From: Alwin Antreich To: pve-user@lists.proxmox.com, Marco Gaiarin Subject: Re: Two nodes asymmetrical cluster, backup, stall of one node. References: In-Reply-To: Content-Type: text/plain; charset=utf-8 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.600 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: PY5VW3QLI2DR3VCOLXV7L3PNCADS63IB X-Message-ID-Hash: PY5VW3QLI2DR3VCOLXV7L3PNCADS63IB X-MailFrom: alwin@antreich.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE user list List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Marco,     "Marco Gaiarin" gaio@lilliput.linux.it – 31. August 2026 um 23:10 >  > I've hit a trouble i cannot understand very well; we have some asymmetrical > clusters, composed of one main server and a 'backup', a little underpowered, server > that act as backup, exporting their fs via NFS. >  > At every backup run (weekly), the backup node got some pressure (this is > expected and common to other clusters), kernel start to complain as: >  > 2026-08-30T00:18:43.125643+02:00 pdpve2 pvescheduler[1474088]: jobs: cfs-lock 'file-jobs_cfg' error: got lock request timeout > 2026-08-30T00:18:43.723075+02:00 pdpve2 pvescheduler[1474087]: replication: cfs-lock 'file-replication_cfg' error: got lock request timeout > [...] > 2026-08-30T00:49:59.269199+02:00 pdpve2 kernel: [481202.580590] INFO: task pvescheduler:1478618 blocked for more than 122 seconds. >  > after that, backup node get 'offline' (from main node), pveproxy respond in > backup node but i canot login (local PAM account or domain ones). >  > If i SSH on the backup node, all seems normal and working, no service > stopped, a little test VM running on it working as expected. >  > Only a note, there what seems a stalled backup job: >  > root@pdpve2:~# ps aux | grep pves[r] > root 1497581 0.0 0.6 308788 110928 ? Ds Aug30 0:00 /usr/bin/perl -T /usr/bin/pvesr prepare-local-job 100-0 --scan local-zfs local-zfs:vm-100-disk-0 --last_sync 1788035400 >  > i've trid to kill it, even with '-9', but whitout success. Processes in D state can't be killed, only a reboot helps.   The pvesr is doing a snapshot sync, which is not involving NFS. Are you running two different things, one replication and a vzdump backup at the same time?  >  >  > If i reboot the node, all come back as normal. >  >  > Someone have some clue? >  >  > If i have a set ov VM on node A and a set of VM on node B, and i run a > backup of all VMs, there's some way to 'serialize' it, eg, do backup of VMs > of node A and after that backup of VM on node B? >  > I suppose that part of the trouble came fron the fact that backup start in > parrallels on both nodes, but node B is also the destination of backup, so > it get 3 time the IO (write from node A, read from node B, write to node B). >  > I can surely setup TWO backup job, but... thanks. That's the solution. Plus add a bandwidth limit to reduce the IO pressure.   Cheers, Alwin