[pve-devel] [PATCH qemu-server 0/6] cloudinit pending behaviour change

[pve-devel] [PATCH qemu-server 0/6] cloudinit pending behaviour change

[Alexandre Derumier]

Hi,

This is an attempt to cleanup current behaviour of cloudinit online changes.

Currently, we setup cloudinit options as pending, until we generate the config drive.

This is not 100% true, because some option like vm name, nic mac address can be changed,
without going to pending, so user can't known if it need to regenerated it.

Also, some can are very difficult to handle, if you hotplug a nic but it's failing,so pending,
then you defined an ipconfig, and then you revert hotplug.
or if you delete a nic, the ipconfig is no more displayed in the gui.


So, instead of setting cloudinit values in pending,
this patch serie copy the current cloudinit config in a new section [special:cloudinit],
when the config drive is generated.
This is only an hint, to allow to display diff between the generated cloudinit
drive, and the current vm config.

A new specific cloudinit config api is added too, merging ipaddrX && netX mac
in same field, and displaying the diff between current and generated config.

Changelog V1:

- use [special:cloudinit] instead [CLOUDINIT] for section
- delete config section on drive removal
- config api: move code to new PVE::QemuServer::Cloudinit::get_pending_config
- config api: add "qm cloudinit pending" cli
- add update api to regenerate drive with 1 api call
- add cloudinit hotplug option

Alexandre Derumier (6):
  cloudinit: add cloudinit section for current generated config.
  generate cloudinit drive on offline plug
  cloudinit: make cloudnit options fastplug
  api2: add cloudinit config api
  api2: add cloudinit_update
  add cloudinit hotplug

 PVE/API2/Qemu.pm            | 117 ++++++++++++++++++++++++++++++++++++
 PVE/CLI/qm.pm               |   2 +
 PVE/QemuServer.pm           | 104 +++++++++++++++++++++-----------
 PVE/QemuServer/Cloudinit.pm | 102 +++++++++++++++++++++++++++++++
 4 files changed, 291 insertions(+), 34 deletions(-)

-- 
2.20.1

[pve-devel] [PATCH qemu-server 1/6] cloudinit: add cloudinit section for current generated config.

[Alexandre Derumier]

Instead using vm pending options for pending cloudinit generated config,

write current generated cloudinit config in a new [special:cloudinit] SECTION.

Currently, some options like vm name, nic mac address can be hotplugged,
so they are not way to know if the cloud-init disk is already updated.
---
 PVE/QemuServer.pm           | 23 ++++++++++++++++++-----
 PVE/QemuServer/Cloudinit.pm | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 5 deletions(-)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 57cfe62..a270104 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -1875,6 +1875,7 @@ sub vmconfig_register_unused_drive {
     if (drive_is_cloudinit($drive)) {
 	eval { PVE::Storage::vdisk_free($storecfg, $drive->{file}) };
 	warn $@ if $@;
+	delete $conf->{cloudinit};
     } elsif (!drive_is_cdrom($drive)) {
 	my $volid = $drive->{file};
 	if (vm_is_volid_owner($storecfg, $vmid, $volid)) {
@@ -2135,6 +2136,7 @@ sub parse_vm_config {
 	digest => Digest::SHA::sha1_hex($raw),
 	snapshots => {},
 	pending => {},
+	cloudinit => {},
     };
 
     $filename =~ m|/qemu-server/(\d+)\.conf$|
@@ -2159,6 +2161,11 @@ sub parse_vm_config {
 	    $descr = undef;
 	    $conf = $res->{$section} = {};
 	    next;
+	} elsif ($line =~ m/^\[special:cloudinit\]\s*$/i) {
+	    $section = 'cloudinit';
+	    $descr = undef;
+	    $conf = $res->{$section} = {};
+	    next;
 
 	} elsif ($line =~ m/^\[([a-z][a-z0-9_\-]+)\]\s*$/i) {
 	    $section = $1;
@@ -2219,7 +2226,6 @@ sub parse_vm_config {
 	    warn "vm $vmid - unable to parse config: $line\n";
 	}
     }
-
     if (defined($descr)) {
 	$descr =~ s/\s+$//;
 	$conf->{description} = $descr;
@@ -2256,7 +2262,7 @@ sub write_vm_config {
 
 	foreach my $key (keys %$cref) {
 	    next if $key eq 'digest' || $key eq 'description' || $key eq 'snapshots' ||
-		$key eq 'snapstate' || $key eq 'pending';
+		$key eq 'snapstate' || $key eq 'pending' || $key eq 'cloudinit';
 	    my $value = $cref->{$key};
 	    if ($key eq 'delete') {
 		die "propertry 'delete' is only allowed in [PENDING]\n"
@@ -2280,8 +2286,10 @@ sub write_vm_config {
 
     &$cleanup_config($conf->{pending}, 1);
 
+    &$cleanup_config($conf->{cloudinit}, 1);
+
     foreach my $snapname (keys %{$conf->{snapshots}}) {
-	die "internal error: snapshot name '$snapname' is forbidden" if lc($snapname) eq 'pending';
+	die "internal error: snapshot name '$snapname' is forbidden" if lc($snapname) =~ m/^(pending|cloudinit)$/; 
 	&$cleanup_config($conf->{snapshots}->{$snapname}, undef, $snapname);
     }
 
@@ -2310,7 +2318,7 @@ sub write_vm_config {
 	}
 
 	foreach my $key (sort keys %$conf) {
-	    next if $key =~ /^(digest|description|pending|snapshots)$/;
+	    next if $key =~ /^(digest|description|pending|cloudinit|snapshots)$/;
 	    $raw .= "$key: $conf->{$key}\n";
 	}
 	return $raw;
@@ -2323,6 +2331,11 @@ sub write_vm_config {
 	$raw .= &$generate_raw_config($conf->{pending}, 1);
     }
 
+    if (scalar(keys %{$conf->{cloudinit}})){
+	$raw .= "\n[special:cloudinit]\n";
+	$raw .= &$generate_raw_config($conf->{cloudinit}, 1);
+    }
+
     foreach my $snapname (sort keys %{$conf->{snapshots}}) {
 	$raw .= "\n[$snapname]\n";
 	$raw .= &$generate_raw_config($conf->{snapshots}->{$snapname});
@@ -4702,9 +4715,9 @@ sub vmconfig_apply_pending {
 	    $conf->{$opt} = delete $conf->{pending}->{$opt};
 	}
     }
-
     # write all changes at once to avoid unnecessary i/o
     PVE::QemuConfig->write_config($vmid, $conf);
+
 }
 
 sub vmconfig_update_net {
diff --git a/PVE/QemuServer/Cloudinit.pm b/PVE/QemuServer/Cloudinit.pm
index c464bf3..f4bf925 100644
--- a/PVE/QemuServer/Cloudinit.pm
+++ b/PVE/QemuServer/Cloudinit.pm
@@ -570,6 +570,38 @@ sub generate_cloudinitconfig {
 
 	$generator->($conf, $vmid, $drive, $volname, $storeid);
     });
+
+    my $cloudinitconf = delete $conf->{cloudinit};
+    $cloudinitconf = {};
+
+    my @cloudinit_opts = keys %{PVE::QemuServer::cloudinit_config_properties()};
+    push @cloudinit_opts, 'name';
+
+    foreach my $opt (@cloudinit_opts) {
+
+	if ($opt =~ m/^ipconfig(\d+)/) {
+	    my $netid = "net$1";
+	    next if !defined($conf->{$netid});
+	    $conf->{cloudinit}->{$netid} = $conf->{$netid};
+	} 
+
+	$conf->{cloudinit}->{$opt} = $conf->{$opt} if $conf->{$opt}; 
+    }
+
+    $conf->{cloudinit}->{name} = "VM$vmid" if !$conf->{cloudinit}->{name};
+
+    foreach my $opt (keys %{$conf}) {
+	if (PVE::QemuServer::is_valid_drivename($opt)) {
+	    my $drive = PVE::QemuServer::parse_drive($opt, $conf->{$opt});
+	    if (PVE::QemuServer::drive_is_cloudinit($drive)) {
+		$conf->{cloudinit}->{$opt} = $conf->{$opt};
+
+	    }
+	}
+    }
+
+    PVE::QemuConfig->write_config($vmid, $conf);
+
 }
 
 sub dump_cloudinit_config {
-- 
2.20.1

[pve-devel] [PATCH qemu-server 2/6] generate cloudinit drive on offline plug

[Alexandre Derumier]

Currently when only generate it at vm start
---
 PVE/QemuServer.pm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index a270104..e28c0dd 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -4702,6 +4702,8 @@ sub vmconfig_apply_pending {
 
     PVE::QemuConfig->cleanup_pending($conf);
 
+    my $generate_cloudnit = undef;
+
     foreach my $opt (keys %{$conf->{pending}}) { # add/change
 	next if $opt eq 'delete'; # just to be sure
 	eval {
@@ -4709,15 +4711,23 @@ sub vmconfig_apply_pending {
 		vmconfig_register_unused_drive($storecfg, $vmid, $conf, parse_drive($opt, $conf->{$opt}))
 	    }
 	};
+
+	if (is_valid_drivename($opt)) {
+	    my $drive = parse_drive($opt, $conf->{pending}->{$opt});
+	    $generate_cloudnit = 1 if drive_is_cloudinit($drive);
+	}
+
 	if (my $err = $@) {
 	    $add_apply_error->($opt, $err);
 	} else {
 	    $conf->{$opt} = delete $conf->{pending}->{$opt};
 	}
     }
+
     # write all changes at once to avoid unnecessary i/o
     PVE::QemuConfig->write_config($vmid, $conf);
 
+    PVE::QemuServer::Cloudinit::generate_cloudinitconfig($conf, $vmid) if $generate_cloudnit;
 }
 
 sub vmconfig_update_net {
-- 
2.20.1

[pve-devel] [PATCH qemu-server 3/6] cloudinit: make cloudnit options fastplug

[Alexandre Derumier]

---
 PVE/QemuServer.pm | 30 +++---------------------------
 1 file changed, 3 insertions(+), 27 deletions(-)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index e28c0dd..bdb5066 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -4456,9 +4456,10 @@ sub vmconfig_hotplug_pending {
 	$errors->{$opt} = "hotplug problem - $msg";
     };
 
+    my @cloudinit_opts = keys %$confdesc_cloudinit;
     my $changes = 0;
     foreach my $opt (keys %{$conf->{pending}}) { # add/change
-	if ($fast_plug_option->{$opt}) {
+	if ($fast_plug_option->{$opt} || grep { $_ eq $opt } @cloudinit_opts) {
 	    $conf->{$opt} = $conf->{pending}->{$opt};
 	    delete $conf->{pending}->{$opt};
 	    $changes = 1;
@@ -4531,31 +4532,6 @@ sub vmconfig_hotplug_pending {
 	}
     }
 
-    my ($apply_pending_cloudinit, $apply_pending_cloudinit_done);
-    $apply_pending_cloudinit = sub {
-	return if $apply_pending_cloudinit_done; # once is enough
-	$apply_pending_cloudinit_done = 1; # once is enough
-
-	my ($key, $value) = @_;
-
-	my @cloudinit_opts = keys %$confdesc_cloudinit;
-	foreach my $opt (keys %{$conf->{pending}}) {
-	    next if !grep { $_ eq $opt } @cloudinit_opts;
-	    $conf->{$opt} = delete $conf->{pending}->{$opt};
-	}
-
-	my $pending_delete_hash = PVE::QemuConfig->parse_pending_delete($conf->{pending}->{delete});
-	foreach my $opt (sort keys %$pending_delete_hash) {
-	    next if !grep { $_ eq $opt } @cloudinit_opts;
-	    PVE::QemuConfig->remove_from_pending_delete($conf, $opt);
-	    delete $conf->{$opt};
-	}
-
-	my $new_conf = { %$conf };
-	$new_conf->{$key} = $value;
-	PVE::QemuServer::Cloudinit::generate_cloudinitconfig($new_conf, $vmid);
-    };
-
     foreach my $opt (keys %{$conf->{pending}}) {
 	next if $selection && !$selection->{$opt};
 	my $value = $conf->{pending}->{$opt};
@@ -4602,7 +4578,7 @@ sub vmconfig_hotplug_pending {
 		# some changes can be done without hotplug
 		my $drive = parse_drive($opt, $value);
 		if (drive_is_cloudinit($drive)) {
-		    &$apply_pending_cloudinit($opt, $value);
+		    PVE::QemuServer::Cloudinit::generate_cloudinitconfig($conf, $vmid);
 		}
 		vmconfig_update_disk($storecfg, $conf, $hotplug_features->{disk},
 				     $vmid, $opt, $value, $arch, $machine_type);
-- 
2.20.1

[pve-devel] [PATCH qemu-server 4/6] api2: add cloudinit config api

[Alexandre Derumier]

---
 PVE/API2/Qemu.pm            | 73 +++++++++++++++++++++++++++++++++++++
 PVE/CLI/qm.pm               |  1 +
 PVE/QemuServer/Cloudinit.pm | 70 +++++++++++++++++++++++++++++++++++
 3 files changed, 144 insertions(+)

diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index ea74c69..b6122fe 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -21,6 +21,7 @@ use PVE::ReplicationConfig;
 use PVE::GuestHelpers;
 use PVE::QemuConfig;
 use PVE::QemuServer;
+use PVE::QemuServer::Cloudinit;
 use PVE::QemuServer::Drive;
 use PVE::QemuServer::CPUConfig;
 use PVE::QemuServer::Monitor qw(mon_cmd);
@@ -1039,6 +1040,78 @@ __PACKAGE__->register_method({
 	return PVE::GuestHelpers::config_with_pending_array($conf, $pending_delete_hash);
    }});
 
+__PACKAGE__->register_method({
+    name => 'cloudinit_pending',
+    path => '{vmid}/cloudinit',
+    method => 'GET',
+    proxyto => 'node',
+    description => "Get the cloudinit configuration with both current and pending values.",
+    permissions => {
+	check => ['perm', '/vms/{vmid}', [ 'VM.Audit' ]],
+    },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    vmid => get_standard_option('pve-vmid', { completion => \&PVE::QemuServer::complete_vmid }),
+	},
+    },
+    returns => {
+	type => "array",
+	items => {
+	    type => "object",
+	    properties => {
+		key => {
+		    description => "Configuration option name.",
+		    type => 'string',
+		},
+		value => {
+		    description => "Current value.",
+		    type => 'string',
+		    optional => 1,
+		},
+		pending => {
+		    description => "Pending value.",
+		    type => 'string',
+		    optional => 1,
+		},
+		delete => {
+		    description => "Indicates a pending delete request if present and not 0. " .
+		                   "The value 2 indicates a force-delete request.",
+		    type => 'integer',
+		    minimum => 0,
+		    maximum => 2,
+		    optional => 1,
+		},
+	    },
+	},
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $vmid = $param->{vmid};
+	my $conf = PVE::QemuConfig->load_config($vmid);
+
+	if( defined($conf->{cipassword}) && 
+	    defined($conf->{cloudinit}->{cipassword}) && 
+	    $conf->{cipassword} ne $conf->{cloudinit}->{cipassword}) {
+	    $conf->{cipassword} = '********** ';
+	} elsif (defined($conf->{cipassword})) {
+	    $conf->{cipassword} = '**********';
+	}
+
+	$conf->{cloudinit}->{cipassword} = '**********' if defined($conf->{cloudinit}->{cipassword});
+
+	my $res = [];
+	my $pending = PVE::QemuServer::Cloudinit::get_pending_config($conf, $vmid);
+
+	foreach my $opt (keys %{$pending}) {
+	    push @$res, $pending->{$opt};
+	}
+
+	return $res;
+   }});
+
 # POST/PUT {vmid}/config implementation
 #
 # The original API used PUT (idempotent) an we assumed that all operations
diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm
index f8972bd..e24b832 100755
--- a/PVE/CLI/qm.pm
+++ b/PVE/CLI/qm.pm
@@ -996,6 +996,7 @@ our $cmddef = {
 		my $data = shift;
 		print "$data\n";
 	    }],
+	pending => [ "PVE::API2::Qemu", 'cloudinit_pending', ['vmid'], { node => $nodename }, \&PVE::GuestHelpers::format_pending ]
     },
 
 };
diff --git a/PVE/QemuServer/Cloudinit.pm b/PVE/QemuServer/Cloudinit.pm
index f4bf925..20cf583 100644
--- a/PVE/QemuServer/Cloudinit.pm
+++ b/PVE/QemuServer/Cloudinit.pm
@@ -629,4 +629,74 @@ sub dump_cloudinit_config {
     }
 }
 
+sub get_pending_config {
+    my ($conf, $vmid) = @_;
+
+    my $newconf = { %{$conf} };
+    my $cloudinit_current = $newconf->{cloudinit};
+    my @cloudinit_opts = keys %{PVE::QemuServer::cloudinit_config_properties()};
+    push @cloudinit_opts, 'name';
+
+    #add cloud-init drive 
+    my $drives = {};
+    PVE::QemuConfig->foreach_volume($newconf, sub {
+	my ($ds, $drive) = @_;
+	$drives->{$ds} = 1 if PVE::QemuServer::drive_is_cloudinit($drive);
+    });
+
+    PVE::QemuConfig->foreach_volume($cloudinit_current, sub {
+	my ($ds, $drive) = @_;
+	$drives->{$ds} = 1 if PVE::QemuServer::drive_is_cloudinit($drive);
+    });
+    foreach my $ds (keys %{$drives}) {
+	push @cloudinit_opts, $ds;
+    }
+
+    $newconf->{name} = "VM$vmid" if !$newconf->{name};
+
+    my $print_net_addr = sub {
+	my ($conf, $opt, $netid) = @_;
+
+	if (defined($conf->{$netid})) {
+
+	    my $net = PVE::QemuServer::parse_net($conf->{$netid});
+	    if (defined($conf->{$opt})) {
+		$conf->{$opt} .= ",macaddr=".$net->{macaddr} if $net->{macaddr};
+	    } else {
+		$conf->{$opt} = "";
+	    }
+	}
+    };
+
+    my $res = {};
+    foreach my $opt (@cloudinit_opts) {
+
+	#add macaddr to ipconfig
+	if ($opt =~ m/^ipconfig(\d+)/) {
+	    my $netid = "net$1";
+	    next if !defined($newconf->{$netid}) && !defined($cloudinit_current->{$netid}) && !defined($newconf->{$opt}) && !defined($cloudinit_current->{$opt} );
+
+	    &$print_net_addr($newconf, $opt, $netid);
+	    &$print_net_addr($cloudinit_current, $opt, $netid);
+	}
+
+	my $item = {
+	    key => $opt,
+	};
+	if ($cloudinit_current->{$opt}) {
+	    $item->{value} = $cloudinit_current->{$opt};
+	    if ($newconf->{$opt}) {
+		$item->{pending} = $newconf->{$opt} if $newconf->{$opt} ne $cloudinit_current->{$opt};
+	    } else {
+		$item->{delete} = 1;
+	    }
+	} else {
+	    $item->{pending} = $newconf->{$opt} if $newconf->{$opt}
+	}
+
+	$res->{$opt} = $item;
+   }
+   return $res;
+}
+
 1;
-- 
2.20.1

[pve-devel] [PATCH qemu-server 5/6] api2: add cloudinit_update

[Alexandre Derumier]

This allow to regenerate the config drive with 1 api call.

This also avoid to delete drive first, and recreate it again.

As it's a readonly drive, we can simply live update it,
and eject/replace it with qemu monitor
---
 PVE/API2/Qemu.pm  | 44 ++++++++++++++++++++++++++++++++++++++++++++
 PVE/CLI/qm.pm     |  3 ++-
 PVE/QemuServer.pm | 28 ++++++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 1 deletion(-)

diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index b6122fe..de2c378 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -1112,6 +1112,50 @@ __PACKAGE__->register_method({
 	return $res;
    }});
 
+__PACKAGE__->register_method({
+    name => 'cloudinit_update',
+    path => '{vmid}/cloudinit',
+    method => 'PUT',
+    protected => 1,
+    proxyto => 'node',
+    description => "Regenerate and change cloudinit config drive.",
+    permissions => {
+	check => ['perm', '/vms/{vmid}', 'VM.Config.Cloudinit', any => 1],
+    },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    vmid => get_standard_option('pve-vmid'),
+	},
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $rpcenv = PVE::RPCEnvironment::get();
+
+	my $authuser = $rpcenv->get_user();
+
+	my $vmid = extract_param($param, 'vmid');
+
+	my $updatefn =  sub {
+
+	    my $conf = PVE::QemuConfig->load_config($vmid);
+
+	    PVE::QemuConfig->check_lock($conf);
+
+	    my $storecfg = PVE::Storage::config();
+
+	    PVE::QemuServer::vmconfig_update_cloudinit_drive($storecfg, $conf, $vmid);
+
+	};
+
+	PVE::QemuConfig->lock_config($vmid, $updatefn);
+
+	return;
+    }});
+
 # POST/PUT {vmid}/config implementation
 #
 # The original API used PUT (idempotent) an we assumed that all operations
diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm
index e24b832..8d594c6 100755
--- a/PVE/CLI/qm.pm
+++ b/PVE/CLI/qm.pm
@@ -996,7 +996,8 @@ our $cmddef = {
 		my $data = shift;
 		print "$data\n";
 	    }],
-	pending => [ "PVE::API2::Qemu", 'cloudinit_pending', ['vmid'], { node => $nodename }, \&PVE::GuestHelpers::format_pending ]
+	pending => [ "PVE::API2::Qemu", 'cloudinit_pending', ['vmid'], { node => $nodename }, \&PVE::GuestHelpers::format_pending ],
+	update => [ "PVE::API2::Qemu", 'cloudinit_update', ['vmid'], { node => $nodename }],
     },
 
 };
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index bdb5066..8518f13 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -4867,6 +4867,34 @@ sub vmconfig_update_disk {
     vm_deviceplug($storecfg, $conf, $vmid, $opt, $drive, $arch, $machine_type);
 }
 
+sub vmconfig_update_cloudinit_drive {
+    my ($storecfg, $conf, $vmid) = @_;
+
+    my $cloudinit_ds = undef;
+    my $cloudinit_drive = undef;
+
+    PVE::QemuConfig->foreach_volume($conf, sub {
+	my ($ds, $drive) = @_;
+	if (PVE::QemuServer::drive_is_cloudinit($drive)) {
+	    $cloudinit_ds = $ds;
+	    $cloudinit_drive = $drive;
+	}
+    });
+
+    return if !$cloudinit_drive;
+
+    PVE::QemuServer::Cloudinit::generate_cloudinitconfig($conf, $vmid);
+    my $running = PVE::QemuServer::check_running($vmid);
+
+    if ($running) {
+	my $path = PVE::Storage::path($storecfg, $cloudinit_drive->{file});
+	if ($path) {
+	    mon_cmd($vmid, "eject", force => JSON::true, id => "$cloudinit_ds");
+	    mon_cmd($vmid, "blockdev-change-medium", id => "$cloudinit_ds", filename => "$path");
+	}
+    }
+}
+
 # called in locked context by incoming migration
 sub vm_migrate_get_nbd_disks {
     my ($storecfg, $conf, $replicated_volumes) = @_;
-- 
2.20.1

[pve-devel] [PATCH qemu-server 6/6] add cloudinit hotplug

[Alexandre Derumier]

This allow to regenerate config drive if pending values exist
when we change vm options.
---
 PVE/QemuServer.pm | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 8518f13..e6b0232 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -287,7 +287,7 @@ my $confdesc = {
     hotplug => {
         optional => 1,
         type => 'string', format => 'pve-hotplug-features',
-        description => "Selectively enable hotplug features. This is a comma separated list of hotplug features: 'network', 'disk', 'cpu', 'memory' and 'usb'. Use '0' to disable hotplug completely. Value '1' is an alias for the default 'network,disk,usb'.",
+        description => "Selectively enable hotplug features. This is a comma separated list of hotplug features: 'network', 'disk', 'cpu', 'memory', 'usb' and 'cloudinit'. Use '0' to disable hotplug completely. Value '1' is an alias for the default 'network,disk,usb'.",
         default => 'network,disk,usb',
     },
     reboot => {
@@ -1298,7 +1298,7 @@ sub parse_hotplug_features {
     $data = $confdesc->{hotplug}->{default} if $data eq '1';
 
     foreach my $feature (PVE::Tools::split_list($data)) {
-	if ($feature =~ m/^(network|disk|cpu|memory|usb)$/) {
+	if ($feature =~ m/^(network|disk|cpu|memory|usb|cloudinit)$/) {
 	    $res->{$1} = 1;
 	} else {
 	    die "invalid hotplug feature '$feature'\n";
@@ -4601,8 +4601,17 @@ sub vmconfig_hotplug_pending {
 	    delete $conf->{pending}->{$opt};
 	}
     }
-
     PVE::QemuConfig->write_config($vmid, $conf);
+
+    if($hotplug_features->{cloudinit}) {
+	my $pending = PVE::QemuServer::Cloudinit::get_pending_config($conf, $vmid);
+	my $regenerate = undef;
+	foreach my $opt (keys %{$pending}) {
+	    my $item = $pending->{$opt};
+	    $regenerate = 1 if defined($item->{delete}) or defined($item->{pending});
+	}
+	PVE::QemuServer::vmconfig_update_cloudinit_drive($storecfg, $conf, $vmid) if $regenerate;
+    }
 }
 
 sub try_deallocate_drive {
-- 
2.20.1

Re: [pve-devel] [PATCH qemu-server 1/6] cloudinit: add cloudinit section for current generated config.

[Mira Limbeck]

Thank you for the patch series and the GUI patches.

Some comments inline.

On 3/28/21 5:11 PM, Alexandre Derumier wrote:
> Instead using vm pending options for pending cloudinit generated config,
>
> write current generated cloudinit config in a new [special:cloudinit] SECTION.
>
> Currently, some options like vm name, nic mac address can be hotplugged,
> so they are not way to know if the cloud-init disk is already updated.
> ---
>   PVE/QemuServer.pm           | 23 ++++++++++++++++++-----
>   PVE/QemuServer/Cloudinit.pm | 32 ++++++++++++++++++++++++++++++++
>   2 files changed, 50 insertions(+), 5 deletions(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 57cfe62..a270104 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -1875,6 +1875,7 @@ sub vmconfig_register_unused_drive {
>       if (drive_is_cloudinit($drive)) {
>   	eval { PVE::Storage::vdisk_free($storecfg, $drive->{file}) };
>   	warn $@ if $@;
> +	delete $conf->{cloudinit};
>       } elsif (!drive_is_cdrom($drive)) {
>   	my $volid = $drive->{file};
>   	if (vm_is_volid_owner($storecfg, $vmid, $volid)) {
> @@ -2135,6 +2136,7 @@ sub parse_vm_config {
>   	digest => Digest::SHA::sha1_hex($raw),
>   	snapshots => {},
>   	pending => {},
> +	cloudinit => {},
>       };
>   
>       $filename =~ m|/qemu-server/(\d+)\.conf$|
> @@ -2159,6 +2161,11 @@ sub parse_vm_config {
>   	    $descr = undef;
>   	    $conf = $res->{$section} = {};
>   	    next;
> +	} elsif ($line =~ m/^\[special:cloudinit\]\s*$/i) {
> +	    $section = 'cloudinit';
> +	    $descr = undef;
> +	    $conf = $res->{$section} = {};
> +	    next;
>   
>   	} elsif ($line =~ m/^\[([a-z][a-z0-9_\-]+)\]\s*$/i) {
>   	    $section = $1;
> @@ -2219,7 +2226,6 @@ sub parse_vm_config {
>   	    warn "vm $vmid - unable to parse config: $line\n";
>   	}
>       }
> -
>       if (defined($descr)) {
>   	$descr =~ s/\s+$//;
>   	$conf->{description} = $descr;
> @@ -2256,7 +2262,7 @@ sub write_vm_config {
>   
>   	foreach my $key (keys %$cref) {
>   	    next if $key eq 'digest' || $key eq 'description' || $key eq 'snapshots' ||
> -		$key eq 'snapstate' || $key eq 'pending';
> +		$key eq 'snapstate' || $key eq 'pending' || $key eq 'cloudinit';
>   	    my $value = $cref->{$key};
>   	    if ($key eq 'delete') {
>   		die "propertry 'delete' is only allowed in [PENDING]\n"
> @@ -2280,8 +2286,10 @@ sub write_vm_config {
>   
>       &$cleanup_config($conf->{pending}, 1);
>   
> +    &$cleanup_config($conf->{cloudinit}, 1);
> +
>       foreach my $snapname (keys %{$conf->{snapshots}}) {
> -	die "internal error: snapshot name '$snapname' is forbidden" if lc($snapname) eq 'pending';
> +	die "internal error: snapshot name '$snapname' is forbidden" if lc($snapname) =~ m/^(pending|cloudinit)$/;

This should be 'special:cloudinit', currently you can't create a 
snapshot with the name 'cloudinit'.

Also there's a trailing whitespace at the end.

>   	&$cleanup_config($conf->{snapshots}->{$snapname}, undef, $snapname);
>       }
>   
> @@ -2310,7 +2318,7 @@ sub write_vm_config {
>   	}
>   
>   	foreach my $key (sort keys %$conf) {
> -	    next if $key =~ /^(digest|description|pending|snapshots)$/;
> +	    next if $key =~ /^(digest|description|pending|cloudinit|snapshots)$/;
>   	    $raw .= "$key: $conf->{$key}\n";
>   	}
>   	return $raw;
> @@ -2323,6 +2331,11 @@ sub write_vm_config {
>   	$raw .= &$generate_raw_config($conf->{pending}, 1);
>       }
>   
> +    if (scalar(keys %{$conf->{cloudinit}})){
> +	$raw .= "\n[special:cloudinit]\n";
> +	$raw .= &$generate_raw_config($conf->{cloudinit}, 1);
> +    }
> +
>       foreach my $snapname (sort keys %{$conf->{snapshots}}) {
>   	$raw .= "\n[$snapname]\n";
>   	$raw .= &$generate_raw_config($conf->{snapshots}->{$snapname});
> @@ -4702,9 +4715,9 @@ sub vmconfig_apply_pending {
>   	    $conf->{$opt} = delete $conf->{pending}->{$opt};
>   	}
>       }
> -
>       # write all changes at once to avoid unnecessary i/o
>       PVE::QemuConfig->write_config($vmid, $conf);
> +
>   }
>   
>   sub vmconfig_update_net {
> diff --git a/PVE/QemuServer/Cloudinit.pm b/PVE/QemuServer/Cloudinit.pm
> index c464bf3..f4bf925 100644
> --- a/PVE/QemuServer/Cloudinit.pm
> +++ b/PVE/QemuServer/Cloudinit.pm
> @@ -570,6 +570,38 @@ sub generate_cloudinitconfig {
>   
>   	$generator->($conf, $vmid, $drive, $volname, $storeid);
>       });
> +
> +    my $cloudinitconf = delete $conf->{cloudinit};
> +    $cloudinitconf = {};
> +
> +    my @cloudinit_opts = keys %{PVE::QemuServer::cloudinit_config_properties()};
> +    push @cloudinit_opts, 'name';
> +
> +    foreach my $opt (@cloudinit_opts) {
> +
> +	if ($opt =~ m/^ipconfig(\d+)/) {
> +	    my $netid = "net$1";
> +	    next if !defined($conf->{$netid});
> +	    $conf->{cloudinit}->{$netid} = $conf->{$netid};
> +	}
trailing whitespace
> +
> +	$conf->{cloudinit}->{$opt} = $conf->{$opt} if $conf->{$opt};
trailing whitespace
> +    }
> +
> +    $conf->{cloudinit}->{name} = "VM$vmid" if !$conf->{cloudinit}->{name};
> +
> +    foreach my $opt (keys %{$conf}) {
> +	if (PVE::QemuServer::is_valid_drivename($opt)) {
> +	    my $drive = PVE::QemuServer::parse_drive($opt, $conf->{$opt});
> +	    if (PVE::QemuServer::drive_is_cloudinit($drive)) {
> +		$conf->{cloudinit}->{$opt} = $conf->{$opt};
> +
> +	    }
> +	}
> +    }
> +
> +    PVE::QemuConfig->write_config($vmid, $conf);
> +
>   }
>   
>   sub dump_cloudinit_config {

Re: [pve-devel] [PATCH qemu-server 4/6] api2: add cloudinit config api

[Mira Limbeck]

Why do you add the macaddress here? I couldn't find anything in this nor 
in the previous patch series explaining why this is done.

On 3/28/21 5:12 PM, Alexandre Derumier wrote:
> ---
>   PVE/API2/Qemu.pm            | 73 +++++++++++++++++++++++++++++++++++++
>   PVE/CLI/qm.pm               |  1 +
>   PVE/QemuServer/Cloudinit.pm | 70 +++++++++++++++++++++++++++++++++++
>   3 files changed, 144 insertions(+)
>
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index ea74c69..b6122fe 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -21,6 +21,7 @@ use PVE::ReplicationConfig;
>   use PVE::GuestHelpers;
>   use PVE::QemuConfig;
>   use PVE::QemuServer;
> +use PVE::QemuServer::Cloudinit;
>   use PVE::QemuServer::Drive;
>   use PVE::QemuServer::CPUConfig;
>   use PVE::QemuServer::Monitor qw(mon_cmd);
> @@ -1039,6 +1040,78 @@ __PACKAGE__->register_method({
>   	return PVE::GuestHelpers::config_with_pending_array($conf, $pending_delete_hash);
>      }});
>   
> +__PACKAGE__->register_method({
> +    name => 'cloudinit_pending',
> +    path => '{vmid}/cloudinit',
> +    method => 'GET',
> +    proxyto => 'node',
> +    description => "Get the cloudinit configuration with both current and pending values.",
> +    permissions => {
> +	check => ['perm', '/vms/{vmid}', [ 'VM.Audit' ]],
> +    },
> +    parameters => {
> +	additionalProperties => 0,
> +	properties => {
> +	    node => get_standard_option('pve-node'),
> +	    vmid => get_standard_option('pve-vmid', { completion => \&PVE::QemuServer::complete_vmid }),
> +	},
> +    },
> +    returns => {
> +	type => "array",
> +	items => {
> +	    type => "object",
> +	    properties => {
> +		key => {
> +		    description => "Configuration option name.",
> +		    type => 'string',
> +		},
> +		value => {
> +		    description => "Current value.",
> +		    type => 'string',
> +		    optional => 1,
> +		},
> +		pending => {
> +		    description => "Pending value.",
> +		    type => 'string',
> +		    optional => 1,
> +		},
> +		delete => {
> +		    description => "Indicates a pending delete request if present and not 0. " .
> +		                   "The value 2 indicates a force-delete request.",
> +		    type => 'integer',
> +		    minimum => 0,
> +		    maximum => 2,
> +		    optional => 1,
> +		},
> +	    },
> +	},
> +    },
> +    code => sub {
> +	my ($param) = @_;
> +
> +	my $vmid = $param->{vmid};
> +	my $conf = PVE::QemuConfig->load_config($vmid);
> +
> +	if( defined($conf->{cipassword}) &&
> +	    defined($conf->{cloudinit}->{cipassword}) &&
> +	    $conf->{cipassword} ne $conf->{cloudinit}->{cipassword}) {
> +	    $conf->{cipassword} = '********** ';
> +	} elsif (defined($conf->{cipassword})) {
> +	    $conf->{cipassword} = '**********';
> +	}
> +
> +	$conf->{cloudinit}->{cipassword} = '**********' if defined($conf->{cloudinit}->{cipassword});
> +
> +	my $res = [];
> +	my $pending = PVE::QemuServer::Cloudinit::get_pending_config($conf, $vmid);
> +
> +	foreach my $opt (keys %{$pending}) {
> +	    push @$res, $pending->{$opt};
> +	}
> +
> +	return $res;
> +   }});
> +
>   # POST/PUT {vmid}/config implementation
>   #
>   # The original API used PUT (idempotent) an we assumed that all operations
> diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm
> index f8972bd..e24b832 100755
> --- a/PVE/CLI/qm.pm
> +++ b/PVE/CLI/qm.pm
> @@ -996,6 +996,7 @@ our $cmddef = {
>   		my $data = shift;
>   		print "$data\n";
>   	    }],
> +	pending => [ "PVE::API2::Qemu", 'cloudinit_pending', ['vmid'], { node => $nodename }, \&PVE::GuestHelpers::format_pending ]
>       },
>   
>   };
> diff --git a/PVE/QemuServer/Cloudinit.pm b/PVE/QemuServer/Cloudinit.pm
> index f4bf925..20cf583 100644
> --- a/PVE/QemuServer/Cloudinit.pm
> +++ b/PVE/QemuServer/Cloudinit.pm
> @@ -629,4 +629,74 @@ sub dump_cloudinit_config {
>       }
>   }
>   
> +sub get_pending_config {
> +    my ($conf, $vmid) = @_;
> +
> +    my $newconf = { %{$conf} };
> +    my $cloudinit_current = $newconf->{cloudinit};
> +    my @cloudinit_opts = keys %{PVE::QemuServer::cloudinit_config_properties()};
> +    push @cloudinit_opts, 'name';
> +
> +    #add cloud-init drive
> +    my $drives = {};
> +    PVE::QemuConfig->foreach_volume($newconf, sub {
> +	my ($ds, $drive) = @_;
> +	$drives->{$ds} = 1 if PVE::QemuServer::drive_is_cloudinit($drive);
> +    });
> +
> +    PVE::QemuConfig->foreach_volume($cloudinit_current, sub {
> +	my ($ds, $drive) = @_;
> +	$drives->{$ds} = 1 if PVE::QemuServer::drive_is_cloudinit($drive);
> +    });
> +    foreach my $ds (keys %{$drives}) {
> +	push @cloudinit_opts, $ds;
> +    }
> +
> +    $newconf->{name} = "VM$vmid" if !$newconf->{name};
> +
> +    my $print_net_addr = sub {
> +	my ($conf, $opt, $netid) = @_;
> +
> +	if (defined($conf->{$netid})) {
> +
> +	    my $net = PVE::QemuServer::parse_net($conf->{$netid});
> +	    if (defined($conf->{$opt})) {
> +		$conf->{$opt} .= ",macaddr=".$net->{macaddr} if $net->{macaddr};
> +	    } else {
> +		$conf->{$opt} = "";
> +	    }
> +	}
> +    };
> +
> +    my $res = {};
> +    foreach my $opt (@cloudinit_opts) {
> +
> +	#add macaddr to ipconfig
> +	if ($opt =~ m/^ipconfig(\d+)/) {
> +	    my $netid = "net$1";
> +	    next if !defined($newconf->{$netid}) && !defined($cloudinit_current->{$netid}) && !defined($newconf->{$opt}) && !defined($cloudinit_current->{$opt} );
> +
> +	    &$print_net_addr($newconf, $opt, $netid);
> +	    &$print_net_addr($cloudinit_current, $opt, $netid);
> +	}
> +
> +	my $item = {
> +	    key => $opt,
> +	};
> +	if ($cloudinit_current->{$opt}) {
> +	    $item->{value} = $cloudinit_current->{$opt};
> +	    if ($newconf->{$opt}) {
> +		$item->{pending} = $newconf->{$opt} if $newconf->{$opt} ne $cloudinit_current->{$opt};
> +	    } else {
> +		$item->{delete} = 1;
> +	    }
> +	} else {
> +	    $item->{pending} = $newconf->{$opt} if $newconf->{$opt}
> +	}
> +
> +	$res->{$opt} = $item;
> +   }
> +   return $res;
> +}
> +
>   1;

Re: [pve-devel] [PATCH qemu-server 4/6] api2: add cloudinit config api

[aderumier]

hi,

> Why do you add the macaddress here? I couldn't find anything in this
> nor 
> in the previous patch series explaining why this is done.

This is mostly to show to user that config drive need to be
regenereted.

cloud-init agent use mac address to map ip to correct interface,

so if you change mac address, you need to reload cloudnit.

Another way could be to show net0 interface, but other attributes like
vlan,bridge,.. 
don't have any effect on cloudnit and don't need any regenerate.


Le mercredi 31 mars 2021 à 16:13 +0200, Mira Limbeck a écrit :
> Why do you add the macaddress here? I couldn't find anything in this
> nor 
> in the previous patch series explaining why this is done.
> 
> On 3/28/21 5:12 PM, Alexandre Derumier wrote:
> > ---
> >   PVE/API2/Qemu.pm            | 73
> > +++++++++++++++++++++++++++++++++++++
> >   PVE/CLI/qm.pm               |  1 +
> >   PVE/QemuServer/Cloudinit.pm | 70
> > +++++++++++++++++++++++++++++++++++
> >   3 files changed, 144 insertions(+)
> > 
> > diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> > index ea74c69..b6122fe 100644
> > --- a/PVE/API2/Qemu.pm
> > +++ b/PVE/API2/Qemu.pm
> > @@ -21,6 +21,7 @@ use PVE::ReplicationConfig;
> >   use PVE::GuestHelpers;
> >   use PVE::QemuConfig;
> >   use PVE::QemuServer;
> > +use PVE::QemuServer::Cloudinit;
> >   use PVE::QemuServer::Drive;
> >   use PVE::QemuServer::CPUConfig;
> >   use PVE::QemuServer::Monitor qw(mon_cmd);
> > @@ -1039,6 +1040,78 @@ __PACKAGE__->register_method({
> >         return PVE::GuestHelpers::config_with_pending_array($conf,
> > $pending_delete_hash);
> >      }});
> >   
> > +__PACKAGE__->register_method({
> > +    name => 'cloudinit_pending',
> > +    path => '{vmid}/cloudinit',
> > +    method => 'GET',
> > +    proxyto => 'node',
> > +    description => "Get the cloudinit configuration with both
> > current and pending values.",
> > +    permissions => {
> > +       check => ['perm', '/vms/{vmid}', [ 'VM.Audit' ]],
> > +    },
> > +    parameters => {
> > +       additionalProperties => 0,
> > +       properties => {
> > +           node => get_standard_option('pve-node'),
> > +           vmid => get_standard_option('pve-vmid', { completion =>
> > \&PVE::QemuServer::complete_vmid }),
> > +       },
> > +    },
> > +    returns => {
> > +       type => "array",
> > +       items => {
> > +           type => "object",
> > +           properties => {
> > +               key => {
> > +                   description => "Configuration option name.",
> > +                   type => 'string',
> > +               },
> > +               value => {
> > +                   description => "Current value.",
> > +                   type => 'string',
> > +                   optional => 1,
> > +               },
> > +               pending => {
> > +                   description => "Pending value.",
> > +                   type => 'string',
> > +                   optional => 1,
> > +               },
> > +               delete => {
> > +                   description => "Indicates a pending delete
> > request if present and not 0. " .
> > +                                  "The value 2 indicates a force-
> > delete request.",
> > +                   type => 'integer',
> > +                   minimum => 0,
> > +                   maximum => 2,
> > +                   optional => 1,
> > +               },
> > +           },
> > +       },
> > +    },
> > +    code => sub {
> > +       my ($param) = @_;
> > +
> > +       my $vmid = $param->{vmid};
> > +       my $conf = PVE::QemuConfig->load_config($vmid);
> > +
> > +       if( defined($conf->{cipassword}) &&
> > +           defined($conf->{cloudinit}->{cipassword}) &&
> > +           $conf->{cipassword} ne $conf->{cloudinit}-
> > >{cipassword}) {
> > +           $conf->{cipassword} = '********** ';
> > +       } elsif (defined($conf->{cipassword})) {
> > +           $conf->{cipassword} = '**********';
> > +       }
> > +
> > +       $conf->{cloudinit}->{cipassword} = '**********' if
> > defined($conf->{cloudinit}->{cipassword});
> > +
> > +       my $res = [];
> > +       my $pending =
> > PVE::QemuServer::Cloudinit::get_pending_config($conf, $vmid);
> > +
> > +       foreach my $opt (keys %{$pending}) {
> > +           push @$res, $pending->{$opt};
> > +       }
> > +
> > +       return $res;
> > +   }});
> > +
> >   # POST/PUT {vmid}/config implementation
> >   #
> >   # The original API used PUT (idempotent) an we assumed that all
> > operations
> > diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm
> > index f8972bd..e24b832 100755
> > --- a/PVE/CLI/qm.pm
> > +++ b/PVE/CLI/qm.pm
> > @@ -996,6 +996,7 @@ our $cmddef = {
> >                 my $data = shift;
> >                 print "$data\n";
> >             }],
> > +       pending => [ "PVE::API2::Qemu", 'cloudinit_pending',
> > ['vmid'], { node => $nodename },
> > \&PVE::GuestHelpers::format_pending ]
> >       },
> >   
> >   };
> > diff --git a/PVE/QemuServer/Cloudinit.pm
> > b/PVE/QemuServer/Cloudinit.pm
> > index f4bf925..20cf583 100644
> > --- a/PVE/QemuServer/Cloudinit.pm
> > +++ b/PVE/QemuServer/Cloudinit.pm
> > @@ -629,4 +629,74 @@ sub dump_cloudinit_config {
> >       }
> >   }
> >   
> > +sub get_pending_config {
> > +    my ($conf, $vmid) = @_;
> > +
> > +    my $newconf = { %{$conf} };
> > +    my $cloudinit_current = $newconf->{cloudinit};
> > +    my @cloudinit_opts = keys
> > %{PVE::QemuServer::cloudinit_config_properties()};
> > +    push @cloudinit_opts, 'name';
> > +
> > +    #add cloud-init drive
> > +    my $drives = {};
> > +    PVE::QemuConfig->foreach_volume($newconf, sub {
> > +       my ($ds, $drive) = @_;
> > +       $drives->{$ds} = 1 if
> > PVE::QemuServer::drive_is_cloudinit($drive);
> > +    });
> > +
> > +    PVE::QemuConfig->foreach_volume($cloudinit_current, sub {
> > +       my ($ds, $drive) = @_;
> > +       $drives->{$ds} = 1 if
> > PVE::QemuServer::drive_is_cloudinit($drive);
> > +    });
> > +    foreach my $ds (keys %{$drives}) {
> > +       push @cloudinit_opts, $ds;
> > +    }
> > +
> > +    $newconf->{name} = "VM$vmid" if !$newconf->{name};
> > +
> > +    my $print_net_addr = sub {
> > +       my ($conf, $opt, $netid) = @_;
> > +
> > +       if (defined($conf->{$netid})) {
> > +
> > +           my $net = PVE::QemuServer::parse_net($conf->{$netid});
> > +           if (defined($conf->{$opt})) {
> > +               $conf->{$opt} .= ",macaddr=".$net->{macaddr} if
> > $net->{macaddr};
> > +           } else {
> > +               $conf->{$opt} = "";
> > +           }
> > +       }
> > +    };
> > +
> > +    my $res = {};
> > +    foreach my $opt (@cloudinit_opts) {
> > +
> > +       #add macaddr to ipconfig
> > +       if ($opt =~ m/^ipconfig(\d+)/) {
> > +           my $netid = "net$1";
> > +           next if !defined($newconf->{$netid}) &&
> > !defined($cloudinit_current->{$netid}) && !defined($newconf-
> > >{$opt}) && !defined($cloudinit_current->{$opt} );
> > +
> > +           &$print_net_addr($newconf, $opt, $netid);
> > +           &$print_net_addr($cloudinit_current, $opt, $netid);
> > +       }
> > +
> > +       my $item = {
> > +           key => $opt,
> > +       };
> > +       if ($cloudinit_current->{$opt}) {
> > +           $item->{value} = $cloudinit_current->{$opt};
> > +           if ($newconf->{$opt}) {
> > +               $item->{pending} = $newconf->{$opt} if $newconf-
> > >{$opt} ne $cloudinit_current->{$opt};
> > +           } else {
> > +               $item->{delete} = 1;
> > +           }
> > +       } else {
> > +           $item->{pending} = $newconf->{$opt} if $newconf->{$opt}
> > +       }
> > +
> > +       $res->{$opt} = $item;
> > +   }
> > +   return $res;
> > +}
> > +
> >   1;
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 

Re: [pve-devel] [PATCH qemu-server 4/6] api2: add cloudinit config api

[aderumier]

Le mercredi 31 mars 2021 à 19:32 +0200, aderumier@odiso.com a écrit :
> hi,
> 
> > Why do you add the macaddress here? I couldn't find anything in
> > this
> > nor 
> > in the previous patch series explaining why this is done.
> 
> This is mostly to show to user that config drive need to be
> regenereted.

and also, if user delete the nic, but not ipconfig, the diff display
the missing mac, so ipconfig is pending, even if user try to regenerate
the drive.

(Currently, if we delete the iface, the ipconfig is not displayed
anymore in gui,  but value is still in vm config file)

Re: [pve-devel] [PATCH qemu-server 1/6] cloudinit: add cloudinit section for current generated config.

[Thomas Lamprecht]

On 28.03.21 17:11, Alexandre Derumier wrote:
> Instead using vm pending options for pending cloudinit generated config,
> 
> write current generated cloudinit config in a new [special:cloudinit] SECTION.
> 
> Currently, some options like vm name, nic mac address can be hotplugged,
> so they are not way to know if the cloud-init disk is already updated.

actually, why isn't the pending section enough for this?

If stuff can be hot plugged then we can do so and if only that changed we
can just remove it from pending, as normally?

Re: [pve-devel] [PATCH qemu-server 1/6] cloudinit: add cloudinit section for current generated config.

[aderumier]

Le jeudi 01 avril 2021 à 10:54 +0200, Thomas Lamprecht a écrit :
> 
> actually, why isn't the pending section enough for this?
> 
> If stuff can be hot plugged then we can do so and if only that
> changed we
> can just remove it from pending, as normally?

Well, for example, if you change the vm name , how to you manage that ?
do you want to keep it as pending  until we regenerate cloudinit drive?

or more complex, if you change the mac address of the vm. (so
unplug/replug the nic)
for the nic, you don't want to keep it as pending, as technically, is
really plugged.

or if you change the storage of the cloudinit drive, currently they a
no way to known if we need to regenerate it.

The main problem is that pending section, is more for pending qemu
change,
not pending cloudinit config drive regeneration.

For me, both are differents.


Currently,it's working well when vm is offline, because we don't have
any pending, and we regenerate the disk at vm startup.


(I'm really looking to use cloudinit for online config changes, like
for containers)

Re: [pve-devel] [PATCH qemu-server 1/6] cloudinit: add cloudinit section for current generated config.

[aderumier]

Maybe, another way,
instead writing [cloudnit:special] section,

we could write this config section inside the drive image directly when
we generate it  (to avoid to reparse format generate config)

so we could read the drive to display the diff in the gui
(we already have an cloudinit dump api, don't seem to be slow)

what do you think about this ?


Le jeudi 01 avril 2021 à 12:22 +0200, aderumier@odiso.com a écrit :
> Le jeudi 01 avril 2021 à 10:54 +0200, Thomas Lamprecht a écrit :
> > 
> > actually, why isn't the pending section enough for this?
> > 
> > If stuff can be hot plugged then we can do so and if only that
> > changed we
> > can just remove it from pending, as normally?
> 
> Well, for example, if you change the vm name , how to you manage that
> ?
> do you want to keep it as pending  until we regenerate cloudinit
> drive?
> 
> or more complex, if you change the mac address of the vm. (so
> unplug/replug the nic)
> for the nic, you don't want to keep it as pending, as technically, is
> really plugged.
> 
> or if you change the storage of the cloudinit drive, currently they a
> no way to known if we need to regenerate it.
> 
> The main problem is that pending section, is more for pending qemu
> change,
> not pending cloudinit config drive regeneration.
> 
> For me, both are differents.
> 
> 
> Currently,it's working well when vm is offline, because we don't have
> any pending, and we regenerate the disk at vm startup.
> 
> 
> (I'm really looking to use cloudinit for online config changes, like
> for containers)
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [pve-devel] [PATCH qemu-server 1/6] cloudinit: add cloudinit section for current generated config.

[alexandre derumier]

Hi,

if you really want to keep the current pending behaviour,

I think the only way is add new pending values,

like "hostname" example , and for ipconfig, add the mac attribute.

so, this should give something like:

name change:  name(pending 
-->fastplug)--->hostname(pending)---regenerate-->hostname

nic hotplug :  net0 (pending)---->net0 (main config 
ok)--->ipconfi0g:...,newmac.(pending)---generate--->ipconfig0

Note that we shouldn't allow revert of ipconfig0. (It's not exposed to 
gui, but we can do revert from command line)

On 02/04/2021 11:22, aderumier@odiso.com wrote:
> Maybe, another way,
> instead writing [cloudnit:special] section,
>
> we could write this config section inside the drive image directly when
> we generate it  (to avoid to reparse format generate config)
>
> so we could read the drive to display the diff in the gui
> (we already have an cloudinit dump api, don't seem to be slow)
>
> what do you think about this ?
>
>
> Le jeudi 01 avril 2021 à 12:22 +0200, aderumier@odiso.com a écrit :
>> Le jeudi 01 avril 2021 à 10:54 +0200, Thomas Lamprecht a écrit :
>>> actually, why isn't the pending section enough for this?
>>>
>>> If stuff can be hot plugged then we can do so and if only that
>>> changed we
>>> can just remove it from pending, as normally?
>> Well, for example, if you change the vm name , how to you manage that
>> ?
>> do you want to keep it as pending  until we regenerate cloudinit
>> drive?
>>
>> or more complex, if you change the mac address of the vm. (so
>> unplug/replug the nic)
>> for the nic, you don't want to keep it as pending, as technically, is
>> really plugged.
>>
>> or if you change the storage of the cloudinit drive, currently they a
>> no way to known if we need to regenerate it.
>>
>> The main problem is that pending section, is more for pending qemu
>> change,
>> not pending cloudinit config drive regeneration.
>>
>> For me, both are differents.
>>
>>
>> Currently,it's working well when vm is offline, because we don't have
>> any pending, and we regenerate the disk at vm startup.
>>
>>
>> (I'm really looking to use cloudinit for online config changes, like
>> for containers)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>