From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id B7422B94BF for ; Sun, 10 Dec 2023 15:50:19 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9DBA1DA5D for ; Sun, 10 Dec 2023 15:49:49 +0100 (CET) Received: from bastionodiso.odiso.net (bastionodiso.odiso.net [IPv6:2a0a:1580:2000::2d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Sun, 10 Dec 2023 15:49:48 +0100 (CET) Received: from kvmformation3.odiso.net (formationkvm3.odiso.net [10.3.94.12]) by bastionodiso.odiso.net (Postfix) with ESMTP id 3FB7B80C1; Sun, 10 Dec 2023 15:49:42 +0100 (CET) Received: by kvmformation3.odiso.net (Postfix, from userid 0) id 2D1001FF408; Sun, 10 Dec 2023 15:49:42 +0100 (CET) From: Alexandre Derumier To: pve-devel@lists.proxmox.com Date: Sun, 10 Dec 2023 15:49:39 +0100 Message-Id: <20231210144940.2031248-1-aderumier@odiso.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.024 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy HEADER_FROM_DIFFERENT_DOMAINS 0.25 From and EnvelopeFrom 2nd level mail domains are different KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com, qemuserver.pm] Subject: [pve-devel] [PATCH qemu-server 0/1] increase qemu max openfiles limit X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Dec 2023 14:50:19 -0000 Hi, The current qemu max openfile limit is 1024. This is really to low it you use ceph storage through librbd. For each disk, qemu process is doing 1 tcp connection to each osd. I have trigger a bug this week, a vm with 6 disk and a cluster with 100 osd, At boot vm was working fine, but after some time, vm begin to do random disk access timeout. The number of connection was around 600~700. Also the qemu monitor/qemu-agent was hanging too. Other users have also reported this bug: https://forum.proxmox.com/threads/vm-qmp-command-failed-vm-qmp-command-query-proxmox-support-failed.90160/#post-613685 https://forum.proxmox.com/threads/qemu-crash-with-vzdump.131603/ https://bugzilla.proxmox.com/show_bug.cgi?id=4507#c1 This patch use prlimit command to increase the limit after vm start. I don't have found a way to increase it in the qemu scope directly with LimitNOFILE. Alternative fix is to increase max openfile globally with if vm is launched through gui: /etc/systemd/system.conf.d/max-open-files.conf [Manager] DefaultLimitNOFILE=524289:524289 if vm is launched through ssh with qm: /etc/security/limits.d/10-max-open-files.conf root - nofile 524289 But maybe users could already have tuned it for containers, so I think it's better to only change limit for qemu process. Alexandre Derumier (1): fix #4507 : increase qemu max openfiles limit PVE/QemuServer.pm | 6 ++++++ 1 file changed, 6 insertions(+) -- 2.39.2