From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id BF6931FF15F for ; Mon, 26 Aug 2024 13:01:19 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D9BA611406; Mon, 26 Aug 2024 13:01:12 +0200 (CEST) To: pve-devel@lists.proxmox.com Date: Mon, 26 Aug 2024 13:00:21 +0200 In-Reply-To: <20240826110030.1744732-1-alexandre.derumier@groupe-cyllene.com> References: <20240826110030.1744732-1-alexandre.derumier@groupe-cyllene.com> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Alexandre Derumier via pve-devel Precedence: list Cc: Alexandre Derumier X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: [pve-devel] [PATCH pve-manager 1/1] pvestatd: lvmqcow2 : extend disk on io-error Content-Type: multipart/mixed; boundary="===============2722703455525637922==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============2722703455525637922== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 89F66C694F for ; Mon, 26 Aug 2024 13:01:11 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2C66310C67 for ; Mon, 26 Aug 2024 13:00:41 +0200 (CEST) Received: from bastiontest.odiso.net (unknown [IPv6:2a0a:1580:2000:6700::14]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 26 Aug 2024 13:00:39 +0200 (CEST) Received: from formationkvm1.odiso.net (unknown [10.11.201.57]) by bastiontest.odiso.net (Postfix) with ESMTP id 371428267D3; Mon, 26 Aug 2024 13:00:33 +0200 (CEST) Received: by formationkvm1.odiso.net (Postfix, from userid 0) id 013C6102037A; Mon, 26 Aug 2024 13:00:31 +0200 (CEST) From: Alexandre Derumier To: pve-devel@lists.proxmox.com Subject: [PATCH pve-manager 1/1] pvestatd: lvmqcow2 : extend disk on io-error Date: Mon, 26 Aug 2024 13:00:21 +0200 Message-Id: <20240826110030.1744732-4-alexandre.derumier@groupe-cyllene.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240826110030.1744732-1-alexandre.derumier@groupe-cyllene.com> References: <20240826110030.1744732-1-alexandre.derumier@groupe-cyllene.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.007 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_NONE 0.1 DMARC none policy HEADER_FROM_DIFFERENT_DOMAINS 0.249 From and EnvelopeFrom 2nd level mail domains are different KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record T_SCC_BODY_TEXT_LINE -0.01 - if the write are really too fast, and the auto extend from qmp event is too slow, the vm could try to write to an qcow2 offset highter than the lvm underlay. In this case, the vm will be paused in "io-error" mode. To fix it, try to extend drive and resume the vm. Signed-off-by: Alexandre Derumier --- PVE/Service/pvestatd.pm | 62 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm index 8661f774..72244451 100755 --- a/PVE/Service/pvestatd.pm +++ b/PVE/Service/pvestatd.pm @@ -230,12 +230,74 @@ sub auto_balloning { } } +sub auto_extend_vm_disk_on_error { + my ($vmstatus) = @_; + + my $storecfg = PVE::Storage::config(); + + foreach my $vmid (keys %$vmstatus) { + my $d = $vmstatus->{$vmid}; + my $status = $d->{qmpstatus} || $d->{status} || 'stopped'; + next if $status ne 'io-error'; + my $resume = undef; + + my $blockstats = PVE::QemuServer::mon_cmd($vmid, "query-blockstats"); + $blockstats = { map { $_->{device} => $_ } $blockstats->@* }; + + my $conf = eval { PVE::QemuConfig->load_config($vmid) }; + if (my $err = $@) { + warn $err; + next; + } + + PVE::QemuConfig->foreach_volume($conf, sub { + my ($ds, $drive) = @_; + + my $volid = $drive->{file}; + return if !$volid; + + my ($sid, $volname) = PVE::Storage::parse_volume_id($volid, 1); + return if !$sid; + + my $scfg = PVE::Storage::storage_config($storecfg, $sid); + return if $scfg->{type} ne 'lvmqcow2'; + + my $blockinfo = PVE::QemuServer::get_block_info($vmid, $ds, $blockstats); + my $wr_highest_offset = $blockinfo->{wr_highest_offset}; + + my $size = PVE::Storage::volume_size_info($storecfg, $volid, 5); + + #if offset is bigger than size, increase lvm size to highest offset + chunksize + if ($wr_highest_offset >= $size) { + my $chunksize = $scfg->{chunksize} // 1024 * 1024 * 1024; + my $newsize = $wr_highest_offset + $chunksize; + syslog('info', "auto extend disk underlay storage of $blockinfo->{deviceid} to $newsize"); + PVE::Storage::volume_resize($storecfg, $volid, $newsize, 1, 1); + my $threshold = compute_write_threshold($newsize); + qemu_block_set_write_threshold($vmid, $blockinfo->{blocknodeid}, $threshold); + } + #if offset is lower, than mean that size has already been increased async but not fast enough + #we just need to resume + $resume = 1; + }); + + if($resume) { + syslog('info', "resume $vmid"); + eval { PVE::QemuServer::vm_resume($vmid, 1, 1); }; + warn $@ if $@; + } + } +} + sub update_qemu_status { my ($status_cfg) = @_; my $ctime = time(); my $vmstatus = PVE::QemuServer::vmstatus(undef, 1); + eval { auto_extend_vm_disk_on_error($vmstatus); }; + syslog('err', "auto extend disk error: $@") if $@; + eval { auto_balloning($vmstatus); }; syslog('err', "auto ballooning error: $@") if $@; -- 2.39.2 --===============2722703455525637922== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============2722703455525637922==--