[pve-devel] cloudinit: question about cloudinit pending values && hostname/mac address changes

[pve-devel] cloudinit: question about cloudinit pending values && hostname/mac address changes

[aderumier]

Hi,

I have some question about cloudinit hotplug pending values.

Currently, when vm is running, we keep cloudinit specific values
(ipconfigX, dns, ssh,...)  in pending until we regenerate image
manually.


But some other change, like vm name (use for hostname), or nic mac
address . (use to match interface in config nodrive format), are not
keeped as pending.


Why don't we simply auto regenerate the cloudinit config drive after
changes? (and don't use pending values like "pending  cdrom
generation").

Anyway, when vm is offline, we don't have pending state at all, and
config drive is generated only after at vmstart.

Also, currently, to regenerated the iso, we need 2 api call,  
1 to remove cdrom ,  1 to replug cdrom with new config.


I really would like to be able to change cloud-init config like lxc, 
simply update values, and get them auto apply.


What do you think about it ?






 

Re: [pve-devel] cloudinit: question about cloudinit pending values && hostname/mac address changes

[Thomas Lamprecht]

On 21.02.21 18:47, aderumier@odiso.com wrote:
> I have some question about cloudinit hotplug pending values.
> 
> Currently, when vm is running, we keep cloudinit specific values
> (ipconfigX, dns, ssh,...)  in pending until we regenerate image
> manually. 
> 
> But some other change, like vm name (use for hostname), or nic mac
> address . (use to match interface in config nodrive format), are not
> keeped as pending.
> 
> Why don't we simply auto regenerate the cloudinit config drive after
> changes? (and don't use pending values like "pending  cdrom
> generation").

IMO OK, wasn't the other stuff done because of some changes cannot be
applied live?

> 
> Anyway, when vm is offline, we don't have pending state at all, and
> config drive is generated only after at vmstart.
> 
> Also, currently, to regenerated the iso, we need 2 api call,  
> 1 to remove cdrom ,  1 to replug cdrom with new config.
> 
> I really would like to be able to change cloud-init config like lxc, 
> simply update values, and get them auto apply.
> 
> What do you think about it ?

Sounds OK to me, without much thinking into regression possibilities.

In general, I'd like to simplify cloud init anyway, IMO the whole
special disk handling just brought us bug after bug with clone, migrate,
...

I'd like to generate the image just in memory (e.g., /run/qemu-server ?)
and just attach it from there (e.g., just using the first free IDE bus
slot, adding new IDE CD ROM devices need reboot anyway, so if it has to
move to another free slot its not a problem).

For backups we already save the config with the state applied, so there
no change is required.

For live migration we'd need to transfer the current state, not much extra
work but needs a few changes.

For live-snapshots we'd need to save the state too (so that processes
which currently have that open do not die if it changed), also a bit of
changes required.

But I think that would simplify this whole thing a lot, and also would
not require the user to add a cloudinit cdrom to the VM, just configure
it and be done.

Re: [pve-devel] cloudinit: question about cloudinit pending values && hostname/mac address changes

[Wolfgang Bumiller]

> On 02/23/2021 9:27 AM Thomas Lamprecht <t.lamprecht@proxmox.com> wrote:
> 
>  
> On 21.02.21 18:47, aderumier@odiso.com wrote:
> > I have some question about cloudinit hotplug pending values.
> > 
> > Currently, when vm is running, we keep cloudinit specific values
> > (ipconfigX, dns, ssh,...)  in pending until we regenerate image
> > manually. 
> > 
> > But some other change, like vm name (use for hostname), or nic mac
> > address . (use to match interface in config nodrive format), are not
> > keeped as pending.
> > 
> > Why don't we simply auto regenerate the cloudinit config drive after
> > changes? (and don't use pending values like "pending  cdrom
> > generation").
> 
> IMO OK, wasn't the other stuff done because of some changes cannot be
> applied live?

Or maybe just an oversight since the VM name used to have no influence
at all before cloud init.
I'm not sure if automatically regenerating the image is such a good idea
if you consider how programs in the guest might react if they're
currently reading from a vanishing drive... (Simply because, you know,
these things tend to not be too failure-resistent ;-) )
This would be different if we used a network-based cloud-init solution,
but that would just "shift" the required effort from the whole state
keeping thomas mentioned below to actually getting this onto a network
interface *per vm* and in a sane way.

But yes, I can honestly also say that if you're changing cloud-init data
while the VM is currently reading it and it just crashes and you have to
hit the reboot button... that's perfectly fine with me actually.

> > 
> > Anyway, when vm is offline, we don't have pending state at all, and
> > config drive is generated only after at vmstart.
> > 
> > Also, currently, to regenerated the iso, we need 2 api call,  
> > 1 to remove cdrom ,  1 to replug cdrom with new config.
> > 
> > I really would like to be able to change cloud-init config like lxc, 
> > simply update values, and get them auto apply.
> > 
> > What do you think about it ?
> 
> Sounds OK to me, without much thinking into regression possibilities.
> 
> In general, I'd like to simplify cloud init anyway, IMO the whole
> special disk handling just brought us bug after bug with clone, migrate,
> ...
> 
> I'd like to generate the image just in memory (e.g., /run/qemu-server ?)
> and just attach it from there (e.g., just using the first free IDE bus
> slot, adding new IDE CD ROM devices need reboot anyway, so if it has to
> move to another free slot its not a problem).
> 
> For backups we already save the config with the state applied, so there
> no change is required.
> 
> For live migration we'd need to transfer the current state, not much extra
> work but needs a few changes.
> 
> For live-snapshots we'd need to save the state too (so that processes
> which currently have that open do not die if it changed), also a bit of
> changes required.
> 
> But I think that would simplify this whole thing a lot, and also would
> not require the user to add a cloudinit cdrom to the VM, just configure
> it and be done.

Re: [pve-devel] cloudinit: question about cloudinit pending values && hostname/mac address changes

[Thomas Lamprecht]

On 23.02.21 10:06, Wolfgang Bumiller wrote:
> 
>> On 02/23/2021 9:27 AM Thomas Lamprecht <t.lamprecht@proxmox.com> wrote:
>>
>>  
>> On 21.02.21 18:47, aderumier@odiso.com wrote:
>>> I have some question about cloudinit hotplug pending values.
>>>
>>> Currently, when vm is running, we keep cloudinit specific values
>>> (ipconfigX, dns, ssh,...)  in pending until we regenerate image
>>> manually. 
>>>
>>> But some other change, like vm name (use for hostname), or nic mac
>>> address . (use to match interface in config nodrive format), are not
>>> keeped as pending.
>>>
>>> Why don't we simply auto regenerate the cloudinit config drive after
>>> changes? (and don't use pending values like "pending  cdrom
>>> generation").
>>
>> IMO OK, wasn't the other stuff done because of some changes cannot be
>> applied live?
> 
> Or maybe just an oversight since the VM name used to have no influence
> at all before cloud init.
> I'm not sure if automatically regenerating the image is such a good idea
> if you consider how programs in the guest might react if they're
> currently reading from a vanishing drive... (Simply because, you know,
> these things tend to not be too failure-resistent ;-) )

normally the CI service reads this only once at startup and then should
wait on events?

Anything basing on a CD ROM device should be able to handle ejects or
inject at any time...

@Alexandre, did you test how good the Cloudinit clients handle this?

> This would be different if we used a network-based cloud-init solution,
> but that would just "shift" the required effort from the whole state
> keeping thomas mentioned below to actually getting this onto a network
> interface *per vm* and in a sane way.
> 
> But yes, I can honestly also say that if you're changing cloud-init data
> while the VM is currently reading it and it just crashes and you have to
> hit the reboot button... that's perfectly fine with me actually.
> 

CI crashing means probably that that change is not applied, not that the
VM is rendered unusable. So you can only win, as at max for applying changes
you have to do the same as you had to do always without such a change
anyway: reboot

Re: [pve-devel] cloudinit: question about cloudinit pending values && hostname/mac address changes

[aderumier]

>>normally the CI service reads this only once at startup and then
>>should
>>wait on events?

>>Anything basing on a CD ROM device should be able to handle ejects or
>>inject at any time...

>>@Alexandre, did you test how good the Cloudinit clients handle this?

I'm able to change the cdrom with new file, with the /dev/cdrom
mounted, without any problem.

About cloud-init, currently, it don't do nothing on config change,
until you restart the service. (so manually, or through an custom udev
rule detecting cdrom change). 

So I think it's pretty safe, until maybe you restart cloudinit service
manually and at same time change the cdrom.

with opennebula opencontext daemon, the udev rule correctly manage
cdrom change, and apply the change.


Maybe could we add a "hotplug: cloudinit" option to enable auto
regeneration ?


I still don't known how to display currently applying state of nic mac
address change. Currently we have only "pending state" (aka pending vm
hardware configuration), but with cloud-init, we have some kind of
"pending cloudnit" state. (aka pending vm guest ok configuration)
It's not really a problem currently, but It's difficult to known if
cloud-init drive need to be regenerated or not.




Le mardi 23 février 2021 à 10:29 +0100, Thomas Lamprecht a écrit :
> 
> normally the CI service reads this only once at startup and then should
> wait on events?
> 
> Anything basing on a CD ROM device should be able to handle ejects or
> inject at any time...
> 
> @Alexandre, did you test how good the Cloudinit clients handle this?

Re: [pve-devel] cloudinit: question about cloudinit pending values && hostname/mac address changes

[aderumier]

Hi,

I just send a small patch, adding a new hotplug option: cloudinit,

to autoregenerate config drive when cloudinit option are updated.

What do you think about it ?  (Like this user can choose the behaviour)

Le mardi 23 février 2021 à 10:29 +0100, Thomas Lamprecht a écrit :
> On 23.02.21 10:06, Wolfgang Bumiller wrote:
> > 
> > > On 02/23/2021 9:27 AM Thomas Lamprecht <t.lamprecht@proxmox.com>
> > > wrote:
> > > 
> > >  
> > > On 21.02.21 18:47, aderumier@odiso.com wrote:
> > > > I have some question about cloudinit hotplug pending values.
> > > > 
> > > > Currently, when vm is running, we keep cloudinit specific
> > > > values
> > > > (ipconfigX, dns, ssh,...)  in pending until we regenerate image
> > > > manually. 
> > > > 
> > > > But some other change, like vm name (use for hostname), or nic
> > > > mac
> > > > address . (use to match interface in config nodrive format),
> > > > are not
> > > > keeped as pending.
> > > > 
> > > > Why don't we simply auto regenerate the cloudinit config drive
> > > > after
> > > > changes? (and don't use pending values like "pending  cdrom
> > > > generation").
> > > 
> > > IMO OK, wasn't the other stuff done because of some changes
> > > cannot be
> > > applied live?
> > 
> > Or maybe just an oversight since the VM name used to have no
> > influence
> > at all before cloud init.
> > I'm not sure if automatically regenerating the image is such a good
> > idea
> > if you consider how programs in the guest might react if they're
> > currently reading from a vanishing drive... (Simply because, you
> > know,
> > these things tend to not be too failure-resistent ;-) )
> 
> normally the CI service reads this only once at startup and then
> should
> wait on events?
> 
> Anything basing on a CD ROM device should be able to handle ejects or
> inject at any time...
> 
> @Alexandre, did you test how good the Cloudinit clients handle this?
> 
> > This would be different if we used a network-based cloud-init
> > solution,
> > but that would just "shift" the required effort from the whole
> > state
> > keeping thomas mentioned below to actually getting this onto a
> > network
> > interface *per vm* and in a sane way.
> > 
> > But yes, I can honestly also say that if you're changing cloud-init
> > data
> > while the VM is currently reading it and it just crashes and you
> > have to
> > hit the reboot button... that's perfectly fine with me actually.
> > 
> 
> CI crashing means probably that that change is not applied, not that
> the
> VM is rendered unusable. So you can only win, as at max for applying
> changes
> you have to do the same as you had to do always without such a change
> anyway: reboot
>