[pve-devel] [PATCH v5 qemu-server 0/9] cloudinit pending behaviour change

[pve-devel] [PATCH v5 qemu-server 0/9] cloudinit pending behaviour change

[Alexandre Derumier]

Hi,

This is an attempt to cleanup current behaviour of cloudinit online changes.

Currently, we setup cloudinit options as pending, until we generate the config drive.

This is not 100% true, because some option like vm name, nic mac address can be changed,
without going to pending, so user can't known if it need to regenerated it.

Also, some can are very difficult to handle, if you hotplug a nic but it's failing,so pending,
then you defined an ipconfig, and then you revert hotplug.
or if you delete a nic, the ipconfig is no more displayed in the gui.


So, instead of setting cloudinit values in pending,
this patch serie copy the current cloudinit config in a new section [special:cloudinit],
when the config drive is generated.
This is only an hint, to allow to display diff between the generated cloudinit
drive, and the current vm config.

A new specific cloudinit config api is added too displaying the diff between current and generated config.

Changelog V1:

- use [special:cloudinit] instead [CLOUDINIT] for section
- delete config section on drive removal
- config api: move code to new PVE::QemuServer::Cloudinit::get_pending_config
- config api: add "qm cloudinit pending" cli
- add update api to regenerate drive with 1 api call
- add cloudinit hotplug option

Changelog v2:

- fix trailing whitespace in first patch
- revert previous "cloudinit" check in snapshot name (":" character is already forbidden)

Changelog v3:

- extract the current conf from cloudinit drive instead write the special cloudinit section

Changelog v4:

- rebase on v2, keep current cloudinit config in vm configuration
- pending api: display mac address change on netX
- cleanup && fix from Fabian comments


Changelog v5:

- move cloudinit fast_plug_option generation outside vmconfig_hotplug_pending
- remove cloudinit section from vm_config api
- vzdump : skip cloudinit section 
- migration: check target node version && forbid migration if too old && cloudinit section exist.

Alexandre Derumier (9):
  cloudinit: add cloudinit section for current generated config.
  generate cloudinit drive on offline plug
  cloudinit: make cloudnit options fastplug
  api2: add cloudinit config api
  api2: add cloudinit_update
  add cloudinit hotplug
  api2: qemu : don't display cloudinit value
  vzdump : skip special:cloudinit section
  migration: test targetnode min version for cloudinit section

 PVE/API2/Qemu.pm            | 112 ++++++++++++++++++++++++++++++++++++
 PVE/CLI/qm.pm               |   2 +
 PVE/QemuMigrate.pm          |  36 +++++++++++-
 PVE/QemuServer.pm           | 102 ++++++++++++++++++++++----------
 PVE/QemuServer/Cloudinit.pm | 109 +++++++++++++++++++++++++++++++++++
 PVE/QemuServer/Helpers.pm   |   1 +
 PVE/VZDump/QemuServer.pm    |   5 +-
 7 files changed, 334 insertions(+), 33 deletions(-)

-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 1/9] cloudinit: add cloudinit section for current generated config.

[Alexandre Derumier]

Instead using vm pending options for pending cloudinit generated config,

write current generated cloudinit config in a new [special:cloudinit] SECTION.

Currently, some options like vm name, nic mac address can be hotplugged,
so they are not way to know if the cloud-init disk is already updated.

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer.pm           | 20 +++++++++++++++++---
 PVE/QemuServer/Cloudinit.pm | 31 +++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 3 deletions(-)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index e9aa248..b282cfd 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -2003,6 +2003,7 @@ sub vmconfig_register_unused_drive {
     if (drive_is_cloudinit($drive)) {
 	eval { PVE::Storage::vdisk_free($storecfg, $drive->{file}) };
 	warn $@ if $@;
+	delete $conf->{cloudinit};
     } elsif (!drive_is_cdrom($drive)) {
 	my $volid = $drive->{file};
 	if (vm_is_volid_owner($storecfg, $vmid, $volid)) {
@@ -2373,6 +2374,7 @@ sub parse_vm_config {
 	digest => Digest::SHA::sha1_hex($raw),
 	snapshots => {},
 	pending => {},
+	cloudinit => {},
     };
 
     my $handle_error = sub {
@@ -2407,6 +2409,11 @@ sub parse_vm_config {
 	    $descr = undef;
 	    $conf = $res->{$section} = {};
 	    next;
+	} elsif ($line =~ m/^\[special:cloudinit\]\s*$/i) {
+	    $section = 'cloudinit';
+	    $descr = undef;
+	    $conf = $res->{$section} = {};
+	    next;
 
 	} elsif ($line =~ m/^\[([a-z][a-z0-9_\-]+)\]\s*$/i) {
 	    $section = $1;
@@ -2504,7 +2511,7 @@ sub write_vm_config {
 
 	foreach my $key (keys %$cref) {
 	    next if $key eq 'digest' || $key eq 'description' || $key eq 'snapshots' ||
-		$key eq 'snapstate' || $key eq 'pending';
+		$key eq 'snapstate' || $key eq 'pending' || $key eq 'cloudinit';
 	    my $value = $cref->{$key};
 	    if ($key eq 'delete') {
 		die "propertry 'delete' is only allowed in [PENDING]\n"
@@ -2528,6 +2535,8 @@ sub write_vm_config {
 
     &$cleanup_config($conf->{pending}, 1);
 
+    &$cleanup_config($conf->{cloudinit}, 1);
+
     foreach my $snapname (keys %{$conf->{snapshots}}) {
 	die "internal error: snapshot name '$snapname' is forbidden" if lc($snapname) eq 'pending';
 	&$cleanup_config($conf->{snapshots}->{$snapname}, undef, $snapname);
@@ -2558,7 +2567,7 @@ sub write_vm_config {
 	}
 
 	foreach my $key (sort keys %$conf) {
-	    next if $key =~ /^(digest|description|pending|snapshots)$/;
+	    next if $key =~ /^(digest|description|pending|cloudinit|snapshots)$/;
 	    $raw .= "$key: $conf->{$key}\n";
 	}
 	return $raw;
@@ -2571,6 +2580,11 @@ sub write_vm_config {
 	$raw .= &$generate_raw_config($conf->{pending}, 1);
     }
 
+    if (scalar(keys %{$conf->{cloudinit}})){
+	$raw .= "\n[special:cloudinit]\n";
+	$raw .= &$generate_raw_config($conf->{cloudinit}, 1);
+    }
+
     foreach my $snapname (sort keys %{$conf->{snapshots}}) {
 	$raw .= "\n[$snapname]\n";
 	$raw .= &$generate_raw_config($conf->{snapshots}->{$snapname});
@@ -5102,9 +5116,9 @@ sub vmconfig_apply_pending {
 	    $conf->{$opt} = delete $conf->{pending}->{$opt};
 	}
     }
-
     # write all changes at once to avoid unnecessary i/o
     PVE::QemuConfig->write_config($vmid, $conf);
+
 }
 
 sub vmconfig_update_net {
diff --git a/PVE/QemuServer/Cloudinit.pm b/PVE/QemuServer/Cloudinit.pm
index b7daa2a..cdaf4e5 100644
--- a/PVE/QemuServer/Cloudinit.pm
+++ b/PVE/QemuServer/Cloudinit.pm
@@ -574,6 +574,37 @@ sub generate_cloudinitconfig {
 
 	$generator->($conf, $vmid, $drive, $volname, $storeid);
     });
+
+    my $cloudinitconf = delete $conf->{cloudinit};
+    $cloudinitconf = {};
+
+    my @cloudinit_opts = keys %{PVE::QemuServer::cloudinit_config_properties()};
+    push @cloudinit_opts, 'name';
+
+    for my $opt (@cloudinit_opts) {
+
+	if ($opt =~ m/^ipconfig(\d+)/) {
+	    my $netid = "net$1";
+	    next if !defined($conf->{$netid});
+	    $conf->{cloudinit}->{$netid} = $conf->{$netid};
+	}
+
+	$conf->{cloudinit}->{$opt} = $conf->{$opt} if $conf->{$opt};
+    }
+
+    $conf->{cloudinit}->{name} = "VM$vmid" if !$conf->{cloudinit}->{name};
+
+    for my $opt (keys %{$conf}) {
+	if (PVE::QemuServer::is_valid_drivename($opt)) {
+	    my $drive = PVE::QemuServer::parse_drive($opt, $conf->{$opt});
+	    if (PVE::QemuServer::drive_is_cloudinit($drive)) {
+		$conf->{cloudinit}->{$opt} = $conf->{$opt};
+	    }
+	}
+    }
+
+    PVE::QemuConfig->write_config($vmid, $conf);
+
 }
 
 sub dump_cloudinit_config {
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 2/9] generate cloudinit drive on offline plug

[Alexandre Derumier]

Currently when only generate it at vm start

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer.pm | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index b282cfd..ac30769 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -5103,6 +5103,8 @@ sub vmconfig_apply_pending {
 
     PVE::QemuConfig->cleanup_pending($conf);
 
+    my $generate_cloudnit = undef;
+
     foreach my $opt (keys %{$conf->{pending}}) { # add/change
 	next if $opt eq 'delete'; # just to be sure
 	eval {
@@ -5110,15 +5112,24 @@ sub vmconfig_apply_pending {
 		vmconfig_register_unused_drive($storecfg, $vmid, $conf, parse_drive($opt, $conf->{$opt}))
 	    }
 	};
+
 	if (my $err = $@) {
 	    $add_apply_error->($opt, $err);
 	} else {
+
+	    if (is_valid_drivename($opt)) {
+		my $drive = parse_drive($opt, $conf->{pending}->{$opt});
+		$generate_cloudnit = 1 if drive_is_cloudinit($drive);
+	    }
+
 	    $conf->{$opt} = delete $conf->{pending}->{$opt};
 	}
     }
+
     # write all changes at once to avoid unnecessary i/o
     PVE::QemuConfig->write_config($vmid, $conf);
 
+    PVE::QemuServer::Cloudinit::generate_cloudinitconfig($conf, $vmid) if $generate_cloudnit;
 }
 
 sub vmconfig_update_net {
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 3/9] cloudinit: make cloudnit options fastplug

[Alexandre Derumier]

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer.pm | 31 +++++--------------------------
 1 file changed, 5 insertions(+), 26 deletions(-)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index ac30769..93ea4cb 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -4832,6 +4832,10 @@ my $fast_plug_option = {
     'tags' => 1,
 };
 
+for my $opt (keys %$confdesc_cloudinit) {
+    $fast_plug_option->{$opt} = 1;
+};
+
 # hotplug changes in [PENDING]
 # $selection hash can be used to only apply specified options, for
 # example: { cores => 1 } (only apply changed 'cores')
@@ -4927,31 +4931,6 @@ sub vmconfig_hotplug_pending {
 	}
     }
 
-    my ($apply_pending_cloudinit, $apply_pending_cloudinit_done);
-    $apply_pending_cloudinit = sub {
-	return if $apply_pending_cloudinit_done; # once is enough
-	$apply_pending_cloudinit_done = 1; # once is enough
-
-	my ($key, $value) = @_;
-
-	my @cloudinit_opts = keys %$confdesc_cloudinit;
-	foreach my $opt (keys %{$conf->{pending}}) {
-	    next if !grep { $_ eq $opt } @cloudinit_opts;
-	    $conf->{$opt} = delete $conf->{pending}->{$opt};
-	}
-
-	my $pending_delete_hash = PVE::QemuConfig->parse_pending_delete($conf->{pending}->{delete});
-	foreach my $opt (sort keys %$pending_delete_hash) {
-	    next if !grep { $_ eq $opt } @cloudinit_opts;
-	    PVE::QemuConfig->remove_from_pending_delete($conf, $opt);
-	    delete $conf->{$opt};
-	}
-
-	my $new_conf = { %$conf };
-	$new_conf->{$key} = $value;
-	PVE::QemuServer::Cloudinit::generate_cloudinitconfig($new_conf, $vmid);
-    };
-
     foreach my $opt (keys %{$conf->{pending}}) {
 	next if $selection && !$selection->{$opt};
 	my $value = $conf->{pending}->{$opt};
@@ -4998,7 +4977,7 @@ sub vmconfig_hotplug_pending {
 		# some changes can be done without hotplug
 		my $drive = parse_drive($opt, $value);
 		if (drive_is_cloudinit($drive)) {
-		    &$apply_pending_cloudinit($opt, $value);
+		    PVE::QemuServer::Cloudinit::generate_cloudinitconfig($conf, $vmid);
 		}
 		vmconfig_update_disk($storecfg, $conf, $hotplug_features->{disk},
 				     $vmid, $opt, $value, $arch, $machine_type);
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 4/9] api2: add cloudinit config api

[Alexandre Derumier]

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/API2/Qemu.pm            | 68 ++++++++++++++++++++++++++++++++
 PVE/CLI/qm.pm               |  1 +
 PVE/QemuServer/Cloudinit.pm | 78 +++++++++++++++++++++++++++++++++++++
 3 files changed, 147 insertions(+)

diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index a824657..20131eb 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -21,6 +21,7 @@ use PVE::ReplicationConfig;
 use PVE::GuestHelpers;
 use PVE::QemuConfig;
 use PVE::QemuServer;
+use PVE::QemuServer::Cloudinit;
 use PVE::QemuServer::CPUConfig;
 use PVE::QemuServer::Drive;
 use PVE::QemuServer::ImportDisk;
@@ -1289,6 +1290,73 @@ __PACKAGE__->register_method({
 	return PVE::GuestHelpers::config_with_pending_array($conf, $pending_delete_hash);
    }});
 
+__PACKAGE__->register_method({
+    name => 'cloudinit_pending',
+    path => '{vmid}/cloudinit',
+    method => 'GET',
+    proxyto => 'node',
+    description => "Get the cloudinit configuration with both current and pending values.",
+    permissions => {
+	check => ['perm', '/vms/{vmid}', [ 'VM.Audit' ]],
+    },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    vmid => get_standard_option('pve-vmid', { completion => \&PVE::QemuServer::complete_vmid }),
+	},
+    },
+    returns => {
+	type => "array",
+	items => {
+	    type => "object",
+	    properties => {
+		key => {
+		    description => "Configuration option name.",
+		    type => 'string',
+		},
+		value => {
+		    description => "Current value.",
+		    type => 'string',
+		    optional => 1,
+		},
+		pending => {
+		    description => "Pending value.",
+		    type => 'string',
+		    optional => 1,
+		},
+		delete => {
+		    description => "Indicates a pending delete request if present and not 0. " .
+		                   "The value 2 indicates a force-delete request.",
+		    type => 'integer',
+		    minimum => 0,
+		    maximum => 2,
+		    optional => 1,
+		},
+	    },
+	},
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $vmid = $param->{vmid};
+	my $conf = PVE::QemuConfig->load_config($vmid);
+
+	if (defined($conf->{cipassword}) &&
+	    defined($conf->{cloudinit}->{cipassword}) &&
+	    $conf->{cipassword} ne $conf->{cloudinit}->{cipassword}) {
+	    $conf->{cipassword} = '********** ';
+	} elsif (defined($conf->{cipassword})) {
+	    $conf->{cipassword} = '**********';
+	}
+
+	$conf->{cloudinit}->{cipassword} = '**********' if defined($conf->{cloudinit}->{cipassword});
+
+	my $res = PVE::QemuServer::Cloudinit::get_pending_config($conf, $vmid);
+
+	return $res;
+   }});
+
 # POST/PUT {vmid}/config implementation
 #
 # The original API used PUT (idempotent) an we assumed that all operations
diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm
index 6a2e161..f7336a7 100755
--- a/PVE/CLI/qm.pm
+++ b/PVE/CLI/qm.pm
@@ -986,6 +986,7 @@ our $cmddef = {
 		my $data = shift;
 		print "$data\n";
 	    }],
+	pending => [ "PVE::API2::Qemu", 'cloudinit_pending', ['vmid'], { node => $nodename }, \&PVE::GuestHelpers::format_pending ]
     },
 
 };
diff --git a/PVE/QemuServer/Cloudinit.pm b/PVE/QemuServer/Cloudinit.pm
index cdaf4e5..2355953 100644
--- a/PVE/QemuServer/Cloudinit.pm
+++ b/PVE/QemuServer/Cloudinit.pm
@@ -7,6 +7,7 @@ use File::Path;
 use Digest::SHA;
 use URI::Escape;
 use MIME::Base64 qw(encode_base64);
+use Storable qw(dclone);
 
 use PVE::Tools qw(run_command file_set_contents);
 use PVE::Storage;
@@ -632,4 +633,81 @@ sub dump_cloudinit_config {
     }
 }
 
+sub get_pending_config {
+    my ($conf, $vmid) = @_;
+
+    my $newconf = dclone($conf);
+
+    my $cloudinit_current = $newconf->{cloudinit};
+    my @cloudinit_opts = keys %{PVE::QemuServer::cloudinit_config_properties()};
+    push @cloudinit_opts, 'name';
+
+    #add cloud-init drive
+    my $drives = {};
+    PVE::QemuConfig->foreach_volume($newconf, sub {
+	my ($ds, $drive) = @_;
+	$drives->{$ds} = 1 if PVE::QemuServer::drive_is_cloudinit($drive);
+    });
+
+    PVE::QemuConfig->foreach_volume($cloudinit_current, sub {
+	my ($ds, $drive) = @_;
+	$drives->{$ds} = 1 if PVE::QemuServer::drive_is_cloudinit($drive);
+    });
+    foreach my $ds (keys %{$drives}) {
+	push @cloudinit_opts, $ds;
+    }
+
+    $newconf->{name} = "VM$vmid" if !$newconf->{name};
+    $cloudinit_current->{name} = "VM$vmid" if !$cloudinit_current->{name};
+
+    #only mac-address is used in cloud-init config. 
+    #We don't want to display other pending net changes.
+    my $print_cloudinit_net = sub {
+	my ($conf, $opt) = @_;
+
+	if (defined($conf->{$opt})) {
+	    my $net = PVE::QemuServer::parse_net($conf->{$opt});
+	    $conf->{$opt} = "macaddr=".$net->{macaddr} if $net->{macaddr};
+	}
+    };
+
+    my $cloudinit_options = {};
+    for my $opt (@cloudinit_opts) {
+	if ($opt =~ m/^ipconfig(\d+)/) {
+	    my $netid = "net$1";
+ 
+	    next if !defined($newconf->{$netid}) && !defined($cloudinit_current->{$netid}) &&
+		    !defined($newconf->{$opt}) && !defined($cloudinit_current->{$opt});
+
+	    &$print_cloudinit_net($newconf, $netid);
+	    &$print_cloudinit_net($cloudinit_current, $netid);
+	    $cloudinit_options->{$netid} = 1;
+	}
+	$cloudinit_options->{$opt} = 1;
+    }
+
+    my $res = [];
+
+    for my $opt (keys %{$cloudinit_options}) {
+
+	my $item = {
+	    key => $opt,
+	};
+	if ($cloudinit_current->{$opt}) {
+	    $item->{value} = $cloudinit_current->{$opt};
+	    if (defined($newconf->{$opt})) {
+		$item->{pending} = $newconf->{$opt} if $newconf->{$opt} ne $cloudinit_current->{$opt};
+	    } else {
+		$item->{delete} = 1;
+	    }
+	} else {
+	    $item->{pending} = $newconf->{$opt} if $newconf->{$opt}
+	}
+
+	push @$res, $item;
+   }
+
+   return $res;
+}
+
 1;
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 5/9] api2: add cloudinit_update

[Alexandre Derumier]

This allow to regenerate the config drive with 1 api call.

This also avoid to delete drive first, and recreate it again.

As it's a readonly drive, we can simply live update it,
and eject/replace it with qemu monitor

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/API2/Qemu.pm  | 43 +++++++++++++++++++++++++++++++++++++++++++
 PVE/CLI/qm.pm     |  3 ++-
 PVE/QemuServer.pm | 28 ++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index 20131eb..1d5b66f 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -1357,6 +1357,49 @@ __PACKAGE__->register_method({
 	return $res;
    }});
 
+__PACKAGE__->register_method({
+    name => 'cloudinit_update',
+    path => '{vmid}/cloudinit',
+    method => 'PUT',
+    protected => 1,
+    proxyto => 'node',
+    description => "Regenerate and change cloudinit config drive.",
+    permissions => {
+	check => ['perm', '/vms/{vmid}', 'VM.Config.Cloudinit'],
+    },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    vmid => get_standard_option('pve-vmid'),
+	},
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $rpcenv = PVE::RPCEnvironment::get();
+
+	my $authuser = $rpcenv->get_user();
+
+	my $vmid = extract_param($param, 'vmid');
+
+	my $updatefn =  sub {
+
+	    my $conf = PVE::QemuConfig->load_config($vmid);
+
+	    PVE::QemuConfig->check_lock($conf);
+
+	    my $storecfg = PVE::Storage::config();
+
+	    PVE::QemuServer::vmconfig_update_cloudinit_drive($storecfg, $conf, $vmid);
+	};
+
+	PVE::QemuConfig->lock_config($vmid, $updatefn);
+
+	return;
+    }});
+
 # POST/PUT {vmid}/config implementation
 #
 # The original API used PUT (idempotent) an we assumed that all operations
diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm
index f7336a7..0818105 100755
--- a/PVE/CLI/qm.pm
+++ b/PVE/CLI/qm.pm
@@ -986,7 +986,8 @@ our $cmddef = {
 		my $data = shift;
 		print "$data\n";
 	    }],
-	pending => [ "PVE::API2::Qemu", 'cloudinit_pending', ['vmid'], { node => $nodename }, \&PVE::GuestHelpers::format_pending ]
+	pending => [ "PVE::API2::Qemu", 'cloudinit_pending', ['vmid'], { node => $nodename }, \&PVE::GuestHelpers::format_pending ],
+	update => [ "PVE::API2::Qemu", 'cloudinit_update', ['vmid'], { node => $nodename }],
     },
 
 };
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 93ea4cb..a38ae3e 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -5295,6 +5295,34 @@ sub vmconfig_update_disk {
     vm_deviceplug($storecfg, $conf, $vmid, $opt, $drive, $arch, $machine_type);
 }
 
+sub vmconfig_update_cloudinit_drive {
+    my ($storecfg, $conf, $vmid) = @_;
+
+    my $cloudinit_ds = undef;
+    my $cloudinit_drive = undef;
+
+    PVE::QemuConfig->foreach_volume($conf, sub {
+	my ($ds, $drive) = @_;
+	if (PVE::QemuServer::drive_is_cloudinit($drive)) {
+	    $cloudinit_ds = $ds;
+	    $cloudinit_drive = $drive;
+	}
+    });
+
+    return if !$cloudinit_drive;
+
+    PVE::QemuServer::Cloudinit::generate_cloudinitconfig($conf, $vmid);
+    my $running = PVE::QemuServer::check_running($vmid);
+
+    if ($running) {
+	my $path = PVE::Storage::path($storecfg, $cloudinit_drive->{file});
+	if ($path) {
+	    mon_cmd($vmid, "eject", force => JSON::true, id => "$cloudinit_ds");
+	    mon_cmd($vmid, "blockdev-change-medium", id => "$cloudinit_ds", filename => "$path");
+	}
+    }
+}
+
 # called in locked context by incoming migration
 sub vm_migrate_get_nbd_disks {
     my ($storecfg, $conf, $replicated_volumes) = @_;
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 6/9] add cloudinit hotplug

[Alexandre Derumier]

This allow to regenerate config drive if pending values exist
when we change vm options.

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer.pm | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index a38ae3e..babc86f 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -303,7 +303,7 @@ my $confdesc = {
 	optional => 1,
 	type => 'string', format => 'pve-hotplug-features',
 	description => "Selectively enable hotplug features. This is a comma separated list of"
-	    ." hotplug features: 'network', 'disk', 'cpu', 'memory' and 'usb'. Use '0' to disable"
+	    ." hotplug features: 'network', 'disk', 'cpu', 'memory', 'usb' and 'cloudinit'. Use '0' to disable"
 	    ." hotplug completely. Using '1' as value is an alias for the default `network,disk,usb`.",
         default => 'network,disk,usb',
     },
@@ -1356,7 +1356,7 @@ sub parse_hotplug_features {
     $data = $confdesc->{hotplug}->{default} if $data eq '1';
 
     foreach my $feature (PVE::Tools::split_list($data)) {
-	if ($feature =~ m/^(network|disk|cpu|memory|usb)$/) {
+	if ($feature =~ m/^(network|disk|cpu|memory|usb|cloudinit)$/) {
 	    $res->{$1} = 1;
 	} else {
 	    die "invalid hotplug feature '$feature'\n";
@@ -5003,8 +5003,16 @@ sub vmconfig_hotplug_pending {
 	    delete $conf->{pending}->{$opt};
 	}
     }
-
     PVE::QemuConfig->write_config($vmid, $conf);
+
+    if($hotplug_features->{cloudinit}) {
+	my $pending = PVE::QemuServer::Cloudinit::get_pending_config($conf, $vmid);
+	my $regenerate = undef;
+	for my $item (@$pending) {
+	    $regenerate = 1 if defined($item->{delete}) or defined($item->{pending});
+	}
+	PVE::QemuServer::vmconfig_update_cloudinit_drive($storecfg, $conf, $vmid) if $regenerate;
+    }
 }
 
 sub try_deallocate_drive {
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 7/9] api2: qemu : don't display cloudinit value

[Alexandre Derumier]

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/API2/Qemu.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index 1d5b66f..57a75fc 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -1227,6 +1227,7 @@ __PACKAGE__->register_method({
 	    $conf = PVE::QemuConfig->load_current_config($param->{vmid}, $param->{current});
 	}
 	$conf->{cipassword} = '**********' if $conf->{cipassword};
+	delete $conf->{cloudinit};
 	return $conf;
 
     }});
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 8/9] vzdump : skip special:cloudinit section

[Alexandre Derumier]

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/VZDump/QemuServer.pm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/PVE/VZDump/QemuServer.pm b/PVE/VZDump/QemuServer.pm
index 7429508..1d38e63 100644
--- a/PVE/VZDump/QemuServer.pm
+++ b/PVE/VZDump/QemuServer.pm
@@ -214,17 +214,20 @@ sub assemble {
 
     my $found_snapshot;
     my $found_pending;
+    my $found_cloudinit;
     while (defined (my $line = <$conffd>)) {
 	next if $line =~ m/^\#vzdump\#/; # just to be sure
 	next if $line =~ m/^\#qmdump\#/; # just to be sure
 	if ($line =~ m/^\[(.*)\]\s*$/) {
 	    if ($1 =~ m/PENDING/i) {
 		$found_pending = 1;
+	    } elsif ($1 =~ m/special:cloudinit/) {
+		$found_cloudinit = 1;
 	    } else {
 		$found_snapshot = 1;
 	    }
 	}
-	next if $found_snapshot || $found_pending; # skip all snapshots and pending changes config data
+	next if $found_snapshot || $found_pending || $found_cloudinit; # skip all snapshots,pending changes and cloudinit config data
 
 	if ($line =~ m/^unused\d+:\s*(\S+)\s*/) {
 	    $self->loginfo("skip unused drive '$1' (not included into backup)");
-- 
2.30.2

[pve-devel] [PATCH v5 qemu-server 9/9] migration: test targetnode min version for cloudinit section

[Alexandre Derumier]

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuMigrate.pm        | 36 +++++++++++++++++++++++++++++++++++-
 PVE/QemuServer/Helpers.pm |  1 +
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm
index d52dc8d..29a03e9 100644
--- a/PVE/QemuMigrate.pm
+++ b/PVE/QemuMigrate.pm
@@ -5,6 +5,7 @@ use warnings;
 
 use IO::File;
 use IPC::Open2;
+use JSON;
 use POSIX qw( WNOHANG );
 use Time::HiRes qw( usleep );
 
@@ -23,7 +24,7 @@ use PVE::Tunnel;
 use PVE::QemuConfig;
 use PVE::QemuServer::CPUConfig;
 use PVE::QemuServer::Drive;
-use PVE::QemuServer::Helpers qw(min_version);
+use PVE::QemuServer::Helpers qw(min_version version_cmp);
 use PVE::QemuServer::Machine;
 use PVE::QemuServer::Monitor qw(mon_cmd);
 use PVE::QemuServer;
@@ -122,6 +123,13 @@ sub prepare {
     # test if VM exists
     my $conf = $self->{vmconf} = PVE::QemuConfig->load_config($vmid);
 
+    my $version = get_node_pvecfg_version($self->{node});
+    my $cloudinit_config = $conf->{cloudinit};
+
+    if(defined($cloudinit_config) && keys %$cloudinit_config && !pvecfg_min_version($version, 7, 2, 4)) {
+	die "target node is too old and don't support new cloudinit format";
+    }
+
     my $repl_conf = PVE::ReplicationConfig->new();
     $self->{replication_jobcfg} = $repl_conf->find_local_replication_job($vmid, $self->{node});
     $self->{is_replicated} = $repl_conf->check_for_existing_jobs($vmid, 1);
@@ -1284,4 +1292,30 @@ sub round_powerof2 {
     return 2 << int(log($_[0]-1)/log(2));
 }
 
+sub get_node_pvecfg_version {
+    my ($node) = @_;
+
+    my $nodes_version_info = PVE::Cluster::get_node_kv('version-info');
+    my $version = undef;
+    if($nodes_version_info->{$node}) {
+	my $version_info_json = $nodes_version_info->{$node};
+	my $version_info = decode_json($version_info_json);
+	$version = $version_info->{version} if $version_info->{version};
+    }
+    return $version;
+}
+
+sub pvecfg_min_version {
+    my ($verstr, $major, $minor, $release) = @_;
+
+    return 0 if !$verstr;
+
+    if ($verstr =~ m/^(\d+)\.(\d+)-(\d+)/) {
+        return 1 if version_cmp($1, $major, $2, $minor, $3, $release) >= 0;
+        return 0;
+    }
+
+    die "internal error: cannot check version of invalid string '$verstr'";
+}
+
 1;
diff --git a/PVE/QemuServer/Helpers.pm b/PVE/QemuServer/Helpers.pm
index c10d842..017a56c 100644
--- a/PVE/QemuServer/Helpers.pm
+++ b/PVE/QemuServer/Helpers.pm
@@ -12,6 +12,7 @@ use base 'Exporter';
 our @EXPORT_OK = qw(
 min_version
 config_aware_timeout
+version_cmp
 );
 
 my $nodename = PVE::INotify::nodename();
-- 
2.30.2

Re: [pve-devel] [PATCH v5 qemu-server 7/9] api2: qemu : don't display cloudinit value

[Fabian Ebner]

Am 16.05.22 um 18:07 schrieb Alexandre Derumier:
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> ---
>  PVE/API2/Qemu.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index 1d5b66f..57a75fc 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -1227,6 +1227,7 @@ __PACKAGE__->register_method({
>  	    $conf = PVE::QemuConfig->load_current_config($param->{vmid}, $param->{current});

It'd be a bit nicer to have load_current_config() not return the
cloudinit section. And because the implementation in AbstractConfig.pm
shouldn't need to know about the cloudinit section, I think we should
override load_current_config() in QemuConfig.pm (calling the parent
method and removing the cloudinit section there).

>  	}
>  	$conf->{cipassword} = '**********' if $conf->{cipassword};
> +	delete $conf->{cloudinit};
>  	return $conf;
>  
>      }});

Re: [pve-devel] [PATCH v5 qemu-server 9/9] migration: test targetnode min version for cloudinit section

[Fabian Ebner]

Am 16.05.22 um 18:07 schrieb Alexandre Derumier:
> @@ -122,6 +123,13 @@ sub prepare {
>      # test if VM exists
>      my $conf = $self->{vmconf} = PVE::QemuConfig->load_config($vmid);
>  
> +    my $version = get_node_pvecfg_version($self->{node});
> +    my $cloudinit_config = $conf->{cloudinit};
> +
> +    if(defined($cloudinit_config) && keys %$cloudinit_config && !pvecfg_min_version($version, 7, 2, 4)) {

Just to note: version here is for pve-manger and needs to be adapted
upon applying. Also requires a dependency bump, so that having new
pve-manager ensures having new qemu-server.

> +	die "target node is too old and don't support new cloudinit format";

s/don't/doesn't/
s/format/section/
missing newline

> +    }
> +
>      my $repl_conf = PVE::ReplicationConfig->new();
>      $self->{replication_jobcfg} = $repl_conf->find_local_replication_job($vmid, $self->{node});
>      $self->{is_replicated} = $repl_conf->check_for_existing_jobs($vmid, 1);
> @@ -1284,4 +1292,30 @@ sub round_powerof2 {
>      return 2 << int(log($_[0]-1)/log(2));
>  }
>  

These two helpers are better added in PVE::QemuServer::Helpers (or could
even be moved to guest-common or pve-cluster together with version_cmp).

> +sub get_node_pvecfg_version {
> +    my ($node) = @_;
> +
> +    my $nodes_version_info = PVE::Cluster::get_node_kv('version-info');

Also passing the node to get_node_kv() avoids needlessly requesting the
info for all nodes.

> +    my $version = undef;
> +    if($nodes_version_info->{$node}) {
> +	my $version_info_json = $nodes_version_info->{$node};
> +	my $version_info = decode_json($version_info_json);
> +	$version = $version_info->{version} if $version_info->{version};
> +    }
> +    return $version;
> +}
> +
> +sub pvecfg_min_version {
> +    my ($verstr, $major, $minor, $release) = @_;
> +
> +    return 0 if !$verstr;
> +
> +    if ($verstr =~ m/^(\d+)\.(\d+)-(\d+)/) {
> +        return 1 if version_cmp($1, $major, $2, $minor, $3, $release) >= 0;
> +        return 0;

Style nit: should be tabs here.

> +    }
> +
> +    die "internal error: cannot check version of invalid string '$verstr'";
> +}
> +
>  1;

Re: [pve-devel] [PATCH v5 qemu-server 0/9] cloudinit pending behaviour change

[Fabian Ebner]

Am 16.05.22 um 18:07 schrieb Alexandre Derumier:
> Changelog v5:
> 
> - move cloudinit fast_plug_option generation outside vmconfig_hotplug_pending
> - remove cloudinit section from vm_config api
> - vzdump : skip cloudinit section 
> - migration: check target node version && forbid migration if too old && cloudinit section exist.
> 
The last three patches should come before the first one, because they
are preparing the introduction of the new section.

The version checking approach works for migration, but with HA stealing
or manual config move, the issue with config parsing is still there. But
that's not too big of an issue IMHO.