[pve-devel] Feature idea: import cloud images as disks, or at VM creation

[pve-devel] Feature idea: import cloud images as disks, or at VM creation

[Gilles Pietri]

Hi,

There's something I'd like to add to the UI, but I'd like some input 
from others, or at least know how useful it may sound.

I'd like to be able to import cloud image from distributions.

My idea would be to add a qemu template type (could be raw to start, but 
really anything that can be imported using qemu-img could do), and use 
those when needed.

It could be through an to import the disk on an existing VM, or at 
creation: instead of choosing a CD-Rom, we'd chose a template to import 
as the main disk, and the size chosen later on would be used to resize it.

What do you think? Is this dumb? Has anyone worked on such an idea?

I'm willing to start exploring adding a template type named Qemu 
template or so, then working on using that to import a disk, then 
improve the creation ui..

Regards,

Gilles

Re: [pve-devel] Feature idea: import cloud images as disks, or at VM creation

[Thomas Lamprecht]

Hi,

On 09.11.21 03:03, Gilles Pietri wrote:
> There's something I'd like to add to the UI, but I'd like some input from others, or at least know how useful it may sound.
> 
> I'd like to be able to import cloud image from distributions.
> 
> My idea would be to add a qemu template type (could be raw to start, but really anything that can be imported using qemu-img could do), and use those when needed.
> 
> It could be through an to import the disk on an existing VM, or at creation: instead of choosing a CD-Rom, we'd chose a template to import as the main disk, and the size chosen later on would be used to resize it.
> 
> What do you think? Is this dumb? Has anyone worked on such an idea?
> 
> I'm willing to start exploring adding a template type named Qemu template or so, then working on using that to import a disk, then improve the creation ui..

FYI, there was already some effort in adding an OVF importer to the API/GUI:

https://lists.proxmox.com/pipermail/pve-devel/2021-June/048564.html
https://lists.proxmox.com/pipermail/pve-devel/2021-June/048563.html

While it stalled a bit but we still plan to pick that up again in the near
future.

cheers,
Thomas

Re: [pve-devel] Feature idea: import cloud images as disks, or at VM creation

[Wolf Noble]

I would think (always dangerous) that the ability to create new VMs from seed images of differing levels of administratively-blessed sources would warrant a few privilege classes to be able to 
(off the top of my head)
- add new image collections
- bless new images
- flag images as ‘preferred’ ‘testing’ ‘sunsetting’ ‘deprecated’ ‘undeployable’
- create new flag-flavors
- add/alter privilege bundles in relationship to flags
- share image collections with another (cluster/customer)
- check for updates to/fetch new existing blessed images
- create new singleton vms from these blessed images
- create an arbitrary set of images to be usable as a privilege construct (( ie users with privilege X may create N vms from group Y ))
- CRUD storage locations for caching of blessed images)

for example:
1) download $cloud-image from $vmfactory
2) add local environment secret sauce
3) spin up a vm from new image and run some scripted automation tasks validating (some stuff still works with new image) security requirements are implemented 
4) flag image as blessed if sniff test doesn’t cause problems
5) remove/reclassify superseded images 


this proto workflow could be automated via CI for acceptance…  and run on a schedule permitting cluster users to be able to receive regular updated vm seeds while maintaining compliance requirements … with minimal administrative overhead 

granted; this is a long way down a theoretical journey … 

i’m just trying to think through what constructs would make sense to ensure they could be implemented tomorrow should it be deemed a good idea…


[= The contents of this message have been written, read, processed, erased, sorted, sniffed, compressed, rewritten, misspelled, overcompensated, lost, found, and most importantly delivered entirely with recycled electrons =]

> On Apr 24, 2023, at 04:01, Fabian Grünbichler <f.gruenbichler@proxmox.com> wrote:
> 
> On April 24, 2023 10:01 am, DERUMIER, Alexandre wrote:
>> I think it could be done with some kind of new naming for this kind of
>> disk,
>> 
>> like "template-....."  in the storage
>> 
>> to match current lxc behaviour.
>> 
>> 
>> I don't think we need to vm template itself inside this, only the disk.
>> 
>> then use could create a vm like
>> 
>> qm create <vmid> --iscsi0:template-.....
> 
> we basically already have this, it's just not yet on the GUI ;)
> 
> $ qm create/set 123 --scsi0 TARGET_STORAGE:0,import-from=SOURCE_STORAGE:VOLUME,other_option=value
> 
> will import an existing volume (or, if highly privileged, arbitrary
> image/block device) to a newly allocated volume on TARGET_STORAGE.
> 
> also mentioned in the docs for cloud-init:
> 
> https://pve.proxmox.com/pve-docs/chapter-qm.html#_preparing_cloud_init_templates
> 
> we haven't fully hashed out yet how to integrate it into the GUI, but
> it's already available on the API and CLI. 
> 
> one part that might still be worth of discussion is whether to add a new
> dir or naming scheme on storages for VM template files like downloaded
> cloud(-init) images, and then on the GUI only offer up those and volumes
> of VM templates as sources (at least by default), instead of *all*
> images accessible to the user.
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 

Re: [pve-devel] Feature idea: import cloud images as disks, or at VM creation

[Fabian Grünbichler]

On April 24, 2023 10:01 am, DERUMIER, Alexandre wrote:
> I think it could be done with some kind of new naming for this kind of
> disk,
> 
> like "template-....."  in the storage
> 
> to match current lxc behaviour.
> 
> 
> I don't think we need to vm template itself inside this, only the disk.
> 
> then use could create a vm like
> 
> qm create <vmid> --iscsi0:template-.....

we basically already have this, it's just not yet on the GUI ;)

$ qm create/set 123 --scsi0 TARGET_STORAGE:0,import-from=SOURCE_STORAGE:VOLUME,other_option=value

will import an existing volume (or, if highly privileged, arbitrary
image/block device) to a newly allocated volume on TARGET_STORAGE.

also mentioned in the docs for cloud-init:

https://pve.proxmox.com/pve-docs/chapter-qm.html#_preparing_cloud_init_templates

we haven't fully hashed out yet how to integrate it into the GUI, but
it's already available on the API and CLI. 

one part that might still be worth of discussion is whether to add a new
dir or naming scheme on storages for VM template files like downloaded
cloud(-init) images, and then on the GUI only offer up those and volumes
of VM templates as sources (at least by default), instead of *all*
images accessible to the user.

Re: [pve-devel] Feature idea: import cloud images as disks, or at VM creation

[DERUMIER, Alexandre]

I think it could be done with some kind of new naming for this kind of
disk,

like "template-....."  in the storage

to match current lxc behaviour.


I don't think we need to vm template itself inside this, only the disk.

then use could create a vm like

qm create <vmid> --iscsi0:template-.....



Maybe some kind of public image repo sync (like turnkey for lxc),
could be implemented too.



Le dimanche 23 avril 2023 à 17:39 -0500, Wolf Noble a écrit :
> 
> I concur that this (or something like it) would definitely make it
> easier to create new VMs….
> 
> what would the CLI / API implementation look like for this? 
> would new canned privilege levels need to exist?
> 
> 
> 
> 
> > On Apr 23, 2023, at 04:18, Gilou via pve-devel
> > <pve-devel@lists.proxmox.com> wrote:
> 
> > Le 09/11/2021 à 03:03, Gilles Pietri a écrit :
> > Hi,
> > There's something I'd like to add to the UI, but I'd like some
> > input from others, or at least know how useful it may sound.
> > I'd like to be able to import cloud image from distributions.
> > My idea would be to add a qemu template type (could be raw to
> > start, but really anything that can be imported using qemu-img
> > could do), and use those when needed.
> > It could be through an to import the disk on an existing VM, or at
> > creation: instead of choosing a CD-Rom, we'd chose a template to
> > import as the main disk, and the size chosen later on would be used
> > to resize it.
> > What do you think? Is this dumb? Has anyone worked on such an idea?
> > I'm willing to start exploring adding a template type named Qemu
> > template or so, then working on using that to import a disk, then
> > improve the creation ui..
> 
> Hi,
> 
> This is an "old" email from 2021, but the need has risen again, so,
> well, let's talk about it, if anybody cares!
> 
> What to do?
> - implement a QEMU disk template (next to ISO Images and Container
> templates?)
> - make it so you can importdisk easily that to a VM? Attach it, set
> it as a bootdisk, resize it?
> - add the option to do all that in the creation flow for a VM?
> - offer to add a cloud-init disk directly?
> - then maybe offer to set username/password/ssh key, or IP settings
> directly in the creation flow, as cloud-init data
> 
> We could even go as far as add in the template content handler a few
> "well-known" base images such as
> https://cloud-images.ubuntu.com/
> https://cloud.debian.org/images/cloud/
> 
> All that would make self service crazily improved when using Proxmox,
> allowing to provide a great experience, starting from "empty" to a
> running VM, allowing us to "duplicate" the cool CT experience in VMs!
> 
> Plus, the base is already there ;)
> Or am I the only one seeing any interest int that?
> 
> Regards,
> 
> Gilou
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
> [= The contents of this message have been written, read, processed,
> erased, sorted, sniffed, compressed, rewritten, misspelled,
> overcompensated, lost, found, and most importantly delivered entirely
> with recycled electrons =]
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Feature idea: import cloud images as disks, or at VM creation

[Wolf Noble]

I concur that this (or something like it) would definitely make it easier to create new VMs….

what would the CLI / API implementation look like for this? 
would new canned privilege levels need to exist?




> On Apr 23, 2023, at 04:18, Gilou via pve-devel <pve-devel@lists.proxmox.com> wrote:

> Le 09/11/2021 à 03:03, Gilles Pietri a écrit :
> Hi,
> There's something I'd like to add to the UI, but I'd like some input from others, or at least know how useful it may sound.
> I'd like to be able to import cloud image from distributions.
> My idea would be to add a qemu template type (could be raw to start, but really anything that can be imported using qemu-img could do), and use those when needed.
> It could be through an to import the disk on an existing VM, or at creation: instead of choosing a CD-Rom, we'd chose a template to import as the main disk, and the size chosen later on would be used to resize it.
> What do you think? Is this dumb? Has anyone worked on such an idea?
> I'm willing to start exploring adding a template type named Qemu template or so, then working on using that to import a disk, then improve the creation ui..

Hi,

This is an "old" email from 2021, but the need has risen again, so, well, let's talk about it, if anybody cares!

What to do?
- implement a QEMU disk template (next to ISO Images and Container templates?)
- make it so you can importdisk easily that to a VM? Attach it, set it as a bootdisk, resize it?
- add the option to do all that in the creation flow for a VM?
- offer to add a cloud-init disk directly?
- then maybe offer to set username/password/ssh key, or IP settings directly in the creation flow, as cloud-init data

We could even go as far as add in the template content handler a few "well-known" base images such as
https://cloud-images.ubuntu.com/
https://cloud.debian.org/images/cloud/

All that would make self service crazily improved when using Proxmox, allowing to provide a great experience, starting from "empty" to a running VM, allowing us to "duplicate" the cool CT experience in VMs!

Plus, the base is already there ;)
Or am I the only one seeing any interest int that?

Regards,

Gilou
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


[= The contents of this message have been written, read, processed, erased, sorted, sniffed, compressed, rewritten, misspelled, overcompensated, lost, found, and most importantly delivered entirely with recycled electrons =]