From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 6C5891FF15C for ; Fri, 17 Oct 2025 13:35:44 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 83E07476; Fri, 17 Oct 2025 13:35:39 +0200 (CEST) To: pve-devel@lists.proxmox.com Date: Fri, 17 Oct 2025 12:25:26 +0100 In-Reply-To: <20251017112539.26471-1-joao.sousa@eurotux.com> References: <20251017112539.26471-1-joao.sousa@eurotux.com> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Tiago Sousa via pve-devel Precedence: list Cc: Tiago Sousa X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: [pve-devel] [PATCH pve-storage 1/4] pvestord: setup new pvestord daemon Content-Type: multipart/mixed; boundary="===============4655790546787902321==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============4655790546787902321== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id AEB41D0F96 for ; Fri, 17 Oct 2025 13:35:37 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1685227D97 for ; Fri, 17 Oct 2025 13:35:07 +0200 (CEST) Received: from eurotux.com (mail.eurotux.com [185.98.249.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 17 Oct 2025 13:35:05 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by eurotux.com (Postfix) with ESMTP id 5AA7930D17BC; Fri, 17 Oct 2025 12:26:03 +0100 (WEST) Authentication-Results: mail.prd.eurotux.pt (amavisd-new); dkim=pass (2048-bit key) reason="pass (just generated, assumed good)" header.d=eurotux.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=eurotux.com; h= content-transfer-encoding:content-type:content-type:mime-version :references:in-reply-to:x-mailer:message-id:date:date:subject :subject:from:from; s=default; t=1760700361; x=1762514762; bh=zy fYd+dENzWYtsXvUPUcP7BegBEQ28Ed/HHrisvpQqk=; b=BZCpQ3nnqf7TreAFVZ QzhoGTYoNwcXt+R7Og6mukGpWZpS0DDxSpDEC0OfLKxwsVI8R3620WKi14+kZ9TW ++hB3zi+v9XunufkLSag7ELWIeyDumg8vmJhn9UTn02pp9jGCJV6Vs/oDU7TM3np W/rSl+1ZnAC6kV2e5OtjTJ9u40nr6UTc3f03yBQ93ZhLDlapBTuKR82vR2/mh7Ry AQJd48VlE5l+1fTLWx1nACTY4ysmX7XvmUt0gMut4hXm/O1mbZXzB+vKyKkn0k+r jm4p+r+aqSbm/p2ESf/DBnt687kuGGXeKIa4po6KD1l7f0JtRzOWAgHKHXrSWhKK 1MHA== X-Virus-Scanned: amavisd-new at mail.prd.eurotux.pt Received: from eurotux.com ([127.0.0.1]) by localhost (mail.prd.eurotux.pt [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7tFDUhYYDdGT; Fri, 17 Oct 2025 12:26:01 +0100 (WEST) Received: from proxmox.example (184.137.90.149.rev.vodafone.pt [149.90.137.184]) (Authenticated sender: joao.sousa@eurotux.com) by eurotux.com (Postfix) with ESMTPSA id 3D52030CFAF0; Fri, 17 Oct 2025 12:26:01 +0100 (WEST) From: Tiago Sousa To: pve-devel@lists.proxmox.com Subject: [PATCH pve-storage 1/4] pvestord: setup new pvestord daemon Date: Fri, 17 Oct 2025 12:25:26 +0100 Message-ID: <20251017112539.26471-2-joao.sousa@eurotux.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251017112539.26471-1-joao.sousa@eurotux.com> References: <20251017112539.26471-1-joao.sousa@eurotux.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.000 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Signed-off-by: Tiago Sousa --- src/Makefile | 1 + src/PVE/Makefile | 1 + src/PVE/Service/Makefile | 10 ++ src/PVE/Service/pvestord.pm | 193 ++++++++++++++++++++++++++++++++++ src/bin/Makefile | 3 + src/bin/pvestord | 24 +++++ src/services/Makefile | 14 +++ src/services/pvestord.service | 15 +++ 8 files changed, 261 insertions(+) create mode 100644 src/PVE/Service/Makefile create mode 100644 src/PVE/Service/pvestord.pm create mode 100755 src/bin/pvestord create mode 100644 src/services/Makefile create mode 100644 src/services/pvestord.service diff --git a/src/Makefile b/src/Makefile index a322f46..09777f2 100644 --- a/src/Makefile +++ b/src/Makefile @@ -9,6 +9,7 @@ all: install: PVE bin udev-rbd $(MAKE) -C bin install $(MAKE) -C PVE install + $(MAKE) -C services install $(MAKE) -C udev-rbd install =20 .PHONY: test diff --git a/src/PVE/Makefile b/src/PVE/Makefile index 9e9f6aa..01ba360 100644 --- a/src/PVE/Makefile +++ b/src/PVE/Makefile @@ -11,6 +11,7 @@ install: make -C API2 install make -C BackupProvider install make -C CLI install + make -C Service install =20 .PHONY: test test: diff --git a/src/PVE/Service/Makefile b/src/PVE/Service/Makefile new file mode 100644 index 0000000..a581f43 --- /dev/null +++ b/src/PVE/Service/Makefile @@ -0,0 +1,10 @@ +SOURCES=3Dpvestord.pm + +all: + +.PHONY: install +install: $(SOURCES) + install -d -m 0755 $(DESTDIR)$(PERLDIR)/PVE/Service + for i in $(SOURCES); do install -D -m 0644 $$i $(DESTDIR)$(PERLDIR)/PVE= /Service/$$i; done + +clean: diff --git a/src/PVE/Service/pvestord.pm b/src/PVE/Service/pvestord.pm new file mode 100644 index 0000000..29fe016 --- /dev/null +++ b/src/PVE/Service/pvestord.pm @@ -0,0 +1,193 @@ +package PVE::Service::pvestord; + +use strict; +use warnings; + +use Time::HiRes qw (gettimeofday); +use PVE::SafeSyslog; +use PVE::Daemon; +use PVE::Cluster qw(cfs_read_file); +use PVE::Storage; +use PVE::QemuConfig; +use PVE::QemuServer; +use PVE::QemuServer::Drive; +use PVE::QemuServer::Blockdev; +use PVE::QemuServer::Helpers; +use PVE::INotify; + +use base qw(PVE::Daemon); + +my $cmdline =3D [$0, @ARGV]; + +my %daemon_options =3D (restart_on_error =3D> 5, stop_wait_time =3D> 15)= ; +my $daemon =3D __PACKAGE__->new('pvestord', $cmdline, %daemon_options); + +my $nodename =3D PVE::INotify::nodename(); + +sub init { + my ($self) =3D @_; + PVE::Cluster::cfs_update(); +} + +my sub get_drive_id { + my ($block_stats, $blockdev_nodename) =3D @_; + foreach my $drive_id (keys %$block_stats) { + my $entry =3D $block_stats->{$drive_id}; + my $file_blockdev =3D $entry->{parent}->{parent}; + return $drive_id + if ($file_blockdev->{'node-name'} eq $blockdev_nodename); + } + return undef; +} + +my sub dequeue { + my ($queue) =3D @_; + PVE::Storage::lock_extend_queue( + sub { + # TODO: This will have to have some sort of mechanism + # to make sure that the element that is removed is the one + # that this node is handling + shift @$queue; + + PVE::Storage::write_extend_queue($queue); + }, + "Could not lock extend queue file", + ); +} + +sub perform_extend { + my $storecfg =3D PVE::Storage::config(); + my $queue =3D PVE::Storage::extend_queue(); + + my $first_extend_request =3D @$queue[0]; + return if !$first_extend_request; + + my ($vmid, $blockdev_nodename) =3D @$first_extend_request; + + my $vmlist =3D PVE::Cluster::get_vmlist(); + my $owner_nodename =3D $vmlist->{ids}->{$vmid}->{node}; + + if ($owner_nodename eq $nodename) { + my $running =3D PVE::QemuServer::Helpers::vm_running_locally($vm= id); + # NOTE: The block device node name is currently generated using = a SHA-256 hash, + # which makes it impossible to reverse-engineer and identify the= original disk. + # As a result, we must rely on `blockstats` to determine which d= isk corresponds + # to a given node name =E2=80=94 but these statistics are only a= vailable when the machine is running. + # Consider updating the `get_node_name()` function to use a reve= rsible encoding + # (e.g., Base64) instead of a SHA-256 digest to simplify disk id= entification. + + my $extend_function =3D sub { + dequeue($queue); + syslog("info", "Processsing extend request $vmid: $blockdev_= nodename\n"); + + my $block_stats =3D PVE::QemuServer::Blockdev::get_block_sta= ts($vmid); + + my $drive_id =3D get_drive_id($block_stats, $blockdev_nodena= me); + if (!$drive_id) { + syslog("err", "Couldn't find drive_id for blockdev $bloc= kdev_nodename"); + return; + } + my $vm_conf =3D PVE::QemuConfig->load_config($vmid); + my $drive =3D PVE::QemuServer::parse_drive($drive_id, $vm_co= nf->{$drive_id}); + my $volid =3D $drive->{file}; + + PVE::QemuServer::Blockdev::underlay_resize( + $storecfg, $vmid, $drive_id, $volid + ); + }; + PVE::QemuConfig->lock_config($vmid, $extend_function); + } +} + +my $next_update =3D 0; +my $cycle =3D 0; +my $restart_request =3D 0; + +my $initial_memory_usage =3D 0; + +# 1 second cycles +my $updatetime =3D 1; + +sub run { + my ($self) =3D @_; + syslog("info", "Running on node $nodename\n"); + + for (;;) { # forever + # get next extend request + $next_update =3D time() + $updatetime; + + if ($cycle) { + my ($ccsec, $cusec) =3D gettimeofday(); + eval { + # syslog('info', "start status update"); + PVE::Cluster::cfs_update(); + perform_extend(); + }; + my $err =3D $@; + + if ($err) { + syslog('err', "status update error: $err"); + } + + my ($ccsec_end, $cusec_end) =3D gettimeofday(); + my $cptime =3D ($ccsec_end - $ccsec) + ($cusec_end - $cusec)= / 1000000; + + syslog('info', sprintf("extend process time (%.3f seconds)",= $cptime)) + if ($cptime > 1); + } + + $cycle++; + + my $mem =3D PVE::ProcFSTools::read_memory_usage(); + my $resident_kb =3D $mem->{resident} / 1024; + + if (!defined($initial_memory_usage) || ($cycle < 10)) { + $initial_memory_usage =3D $resident_kb; + } else { + my $diff =3D $resident_kb - $initial_memory_usage; + if ($diff > 15 * 1024) { + syslog( + 'info', + "restarting server after $cycle cycles to " + . "reduce memory usage (free $resident_kb ($diff= ) KB)", + ); + $self->restart_daemon(); + } + } + + my $wcount =3D 0; + while ( + (time() < $next_update) + && ($wcount < $updatetime) + && # protect against time wrap + !$restart_request + ) { + $wcount++; + sleep(1); + } + + $self->restart_daemon() if $restart_request; + } +} + +sub shutdown { + my ($self) =3D @_; + + syslog('info', "server closing"); + + $self->exit_daemon(0); +} + +$daemon->register_start_command(); +$daemon->register_restart_command(1); +$daemon->register_stop_command(); +$daemon->register_status_command(); + +our $cmddef =3D { + start =3D> [__PACKAGE__, 'start', []], + restart =3D> [__PACKAGE__, 'restart', []], + stop =3D> [__PACKAGE__, 'stop', []], + status =3D> [__PACKAGE__, 'status', [], undef, sub { print shift . "= \n"; }], +}; + +1; diff --git a/src/bin/Makefile b/src/bin/Makefile index 2e0a080..206c35b 100644 --- a/src/bin/Makefile +++ b/src/bin/Makefile @@ -1,5 +1,6 @@ DESTDIR=3D PREFIX=3D/usr +BINDIR=3D$(PREFIX)/bin SBINDIR=3D$(PREFIX)/sbin MANDIR=3D$(PREFIX)/share/man MAN1DIR=3D$(MANDIR)/man1/ @@ -30,6 +31,8 @@ install: pvesm.1 pvesm.bash-completion pvesm.zsh-comple= tion gzip -9 -n $(DESTDIR)$(MAN1DIR)/pvesm.1 install -m 0644 -D pvesm.bash-completion $(DESTDIR)$(BASHCOMPLDIR)/pves= m install -m 0644 -D pvesm.zsh-completion $(DESTDIR)$(ZSHCOMPLDIR)/_pvesm + install -d $(DESTDIR)$(BINDIR) + install -m 0755 pvestord $(DESTDIR)$(BINDIR) =20 .PHONY: clean clean: diff --git a/src/bin/pvestord b/src/bin/pvestord new file mode 100755 index 0000000..e88a0b3 --- /dev/null +++ b/src/bin/pvestord @@ -0,0 +1,24 @@ +#!/usr/bin/perl + +use strict; +use warnings; + +use PVE::INotify; +use PVE::RPCEnvironment; +use PVE::SafeSyslog; +use PVE::Service::pvestord; + +$SIG{'__WARN__'} =3D sub { + my $err =3D $@; + my $t =3D $_[0]; + chomp $t; + print STDERR "$t\n"; + syslog('warning', "%s", $t); + $@ =3D $err; +}; + +my $prepare =3D sub { + +}; + +PVE::Service::pvestord->run_cli_handler(prepare =3D> $prepare); diff --git a/src/services/Makefile b/src/services/Makefile new file mode 100644 index 0000000..98db674 --- /dev/null +++ b/src/services/Makefile @@ -0,0 +1,14 @@ +SERVICEDIR=3D$(DESTDIR)/usr/lib/systemd/system + +all: + +SERVICES=3D pvestord.service + +.PHONY: install +install: $(SERVICES) + install -d $(SERVICEDIR) + install -m 0644 $(SERVICES) $(SERVICEDIR) + +.PHONY: clean +clean: + rm -rf *~ diff --git a/src/services/pvestord.service b/src/services/pvestord.servic= e new file mode 100644 index 0000000..310fa91 --- /dev/null +++ b/src/services/pvestord.service @@ -0,0 +1,15 @@ +[Unit] +Description=3DPVE Storage Monitor Daemon +ConditionPathExists=3D/usr/bin/pvestord +Wants=3Dpve-cluster.service +After=3Dpve-cluster.service + +[Service] +ExecStart=3D/usr/bin/pvestord start +ExecStop=3D/usr/bin/pvestord stop +ExecReload=3D/usr/bin/pvestord restart +PIDFile=3D/run/pvestord.pid +Type=3Dforking + +[Install] +WantedBy=3Dmulti-user.target --=20 2.47.3 --===============4655790546787902321== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============4655790546787902321==--