From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>,
"t.lamprecht@proxmox.com" <t.lamprecht@proxmox.com>,
"s.hanreich@proxmox.com" <s.hanreich@proxmox.com>
Subject: Re: [pve-devel] [RFC cluster/manager/network 0/6] Add support for DHCP servers to SDN
Date: Tue, 26 Sep 2023 13:07:18 +0000 [thread overview]
Message-ID: <cef49dd992e40135c64ffef0e7f9f572784900bc.camel@groupe-cyllene.com> (raw)
In-Reply-To: <3105e957-706e-d512-36f5-6ba558b347ef@proxmox.com>
Le mardi 26 septembre 2023 à 13:20 +0200, Stefan Hanreich a écrit :
> On 9/20/23 23:48, DERUMIER, Alexandre wrote:
> > Finally, It's not so easy without writing ip on proxmox side (in vm
> > config or somewhere else), because to retrieve a reserved ip from
> > external ipam when vm start, we need to lookup maybe from mac
> > address,
> > maybe from hostname of the vm, or maybe some custom attributes, but
> > not
> > all ipams accept same attributes.
> >
> > (at least phpipam && netbox don't support all features, or not
> > easyly.
> > Netbox for example, for macaddress need to register the full vm
> > object
> > && interfaces + mac + mapping to ip, Phpipam is a single ip object
> > with mac as attribute).
>
> Yes, I think so as well. It also would make us dependent on external
> systems, which might not always be up and would create an additional
> hurdle for setting things up. Having our own solution for this seems
> preferable imo. We can still provide integrations with netbox /
> phpipam
> so they can take over from our small IPAM if they implement the
> features
> we need.
>
> I'll take a closer look at netbox, since I was under the impression
> that
> they should support this - although it's been awhile since I played
> around with it. Not sure about phpIPAM, but I wasn't too stoked on
> using
> it anyway after browsing their source code for a bit.
>
> > So I think the best way is still to write the ip into the vm
> > config,
> > this allow to inject already reserved ip in dhcp at vm
> > start/migrate
> > without need to call the ipam (also avoid start problem is ipam
> > server
> > is down).
> >
> > and this allow to use it for firewall ipfilter, I see a usecase for
> > sdn
> > vxlan too or special /32 route injection)
> >
>
> Yes, I think so as well, although we would need to take care of
> proper
> synchronization between Configs and IPAM. If this diverges for
> whatever
> reason we will run into trouble. Of course, this *should* never
> happen
> when properly implemented.
>
> Another option I thought about would be storing a VMID -> IP mapping
> in
> the (pve) IPAM itself. This would have the upside of having a
> centralized storage and single source of truth without having to
> maintain two different locations where we store the IP.
>
> Though it would also be a bit more intransparent to the user if we
> don't
> expose it somewhere in the UI.
>
> This would have the downside that starting VMs is an issue when the
> IPAM
> is down. While using the pve IPAM in a cluster (or a single node) I
> can
> see this being alright, since you need quorum to start a VM. As long
> as
> you have a quorate cluster the pve IPAM *should* be available as
> well.
>
> In the case of using phpIPAM or netbox this is an issue we would need
> to
> think about.
>
Yes, this is my main concern, as it'll be my case in production, as I
managing multiple clusters, on differents location, with subnets
sharing.
for me, it's ok if ipam is down when allocating a new ip or vm.
But for vm start/stop, I think we should have at minimum some cache
somewhere. (I'm think about a disaster recovery or big network problem,
where you want to fast restart all vms without need to call the ipam).
Maybe a way, could be to use the local pve ipam, as a local mirror of
the external ipam ? (and don't store ip in vm config, but only in
pve ipam, the source of truth)
vm with ipam=auto, external ipam is configured:
- if (ip_exist(pve_ipam)) {
return ip
} elsif (ip_exist(external_ipam)) {
return ip
} else {
ip = findnextfreeip(external_ipam)
sync_pve_ipam(ip)
return ip
}
I have see this in vmware or openstack/opennebula with external ipam,
I don't remember exactly.
> > I just need some protections for snapshot, but nothing too
> > difficult,
> > but we really need to avoid to try to manage in ipam multiple
> > version/snapshot of ip entry for a vm.
> > I had tried 2years ago, it was really painful to handle this in
> > differents ipam.
> > So maybe the best way is to forbid to change ip address when a
> > snapshot
> > already exist.
>
> Yes, it might be just the best way to check on restore if the IP is
> the
> same or at least currently available and otherwise just get a new IP
> from IPAM automatically (maybe with a warning).
>
> On the other hand, this should not be an issue when storing the VMID
> ->
> IP mapping centralized somewhere, since we then can just rely on the
> IP
> being stored there. Of course this would exclude the DHCP/IP setting
> from the snapshot which can be good or bad I'd say (depending on the
> use
> case).
>
> > I think we could implement ipam call like:
> >
> >
> > create vm or add a new nic -->
> > -----------------------------
> > qm create ... -net0
> > bridge=vnet,....,ip=(auto|192.168.0.1|dynamic),ip6=(..)
> >
> >
> > auto : search a free ip in ipam. write the ip address in net0:
> > ...,ip=
> > ip field
> >
> > 192.168.0.1: check if ip is free in ipam && register ip in ipam.
> > write
> > the ip in ip field.
> >
> >
> > dynamic: write "ephemeral" in net0: ....,ip=ephemeral (This is a
> > dynamic ip registered at vm start, and release at vm stop)
>
> Sounds good to me.
>
> >
> > vm start
> > ---------
> > - if ip=ephemeral, find && register a free ip in ipam, write it in
> > vm
> > net0: ...,ip=192.168.0.10[E] . (maybe with a special flag [E] to
> > indicate it's ephemeral)
> > - read ip from vm config && inject in dhcp
>
> Maybe we can even get away with setting the IP in the DHCP config as
> soon as we set it in the VM configuration, as long as it is not
> ephemeral, thus avoiding the need for having to do it while starting
> VMs?
>
> > vm_stop
> > -------
> > if ip is ephemeral (netX: ip=192.168.0.10[E]), delete ip from
> > ipam,
> > set ip=ephemeral in vm config
> >
> >
> > vm_destroy or nic remove/unplug
> > -------------------------
> > if netX: ...,ip=192.168.0.10 , remove ip from ipam
> >
> >
> >
> > nic update when vm is running:
> > ------------------------------
> > if ip is defined : netX: ip=192.168.0.10, we don't allow bridge
> > change
> > or ip change, as vm is not notified about theses changes, and still
> > use
> > old ip.
> >
> > We can allow nic hot-unplug && hotplug. (guest os will remove the
> > ip on
> > nic removal, and will call dhcp again on nic hotplug)
>
> Yes, I think so as well. Maybe we could give the option to change the
> IP
> together with a forced reboot and a warning like 'When changing the
> IP
> this VM will get rebooted' as a quality of life feature?
>
> >
> > nic hotplug with ip=auto:
> > -------------------------
> >
> > --> add nic in pending state ----> find ip in ipam && write it in
> > pending ---> do the hotplug in qemu.
> >
> > We need to handle the config revert to remove ip from ipam if the
> > nic
> > hotplug is blocked in pending state(I never see this case until os
> > don't have pci_hotplug module loaded, but it's better to be
> > carefull )
> >
>
> Yes - defensive programming is always good!
>
> > The ipam modules (internal pve, phpipam,netbox) are already for
> > this, I
> > think it shouldn't be too difficult.
> >
> > dnsmasq seem to have a reservation file option, where we can
> > dynamically add ip-mac without need to reload it.
> >
> > I'll try it, re-using your current dnsmasq patches.
>
> Since you want to take a shot at implementing it, is there anything I
> could help you with? I'd have some resources now for taking a shot at
> this as well.
>
I'm a bit busy currently on other stuff and I would like to finish them
first.
So if you have a little bit time to work on this, it could be great :)
I have send some patches in 2021 for ipam integration in qemu/lxc, if
you want to take some inspiration. (without the ip in the vm config, it
should be a lot easier)
> It would also be interesting to improve and add some features to our
> built-in IPAM, maybe even add the VMID -> IP mapping functionality
> I've
> touched upon earlier. It would also be interesting to be able to
> expose
> some of this information to the frontend, so users have an overview
> of
> currently leased IPs in the frontend - what do you think?
>
Yes,admin should be able to see allocated ip. (like a real ipam).
I was thinking about other stuff for later, but maybe it could be great
for an admin to be able to reserve ips and put them in a pool.
Then user could choose ip from this pool.
(Usecase is public ip addresses, where a customer could buy some of
them,
then allocated them like he want)
> Would it also make sense to set IPSet entries for VMs, so they are
> only
> allowed to use the IPs we dedicate to them? This would be a decent
> safeguard for preventing issues down the line.
>
> Additionally it would be interesting to automatically create Aliases
> for
> VNets/VMs in the Firewall configuration - what do you think? If we
> add
> VMs as Aliases, we would have to recompile the iptables on every IP
> change. For this feature it would make sense to be able to set names
> /
> comments on VNets, so we can reference them this way. What do you
> think?
a Big yes ! (as I'm already doing this manually in production ;)
>
next prev parent reply other threads:[~2023-09-26 13:08 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-08 13:42 Stefan Hanreich
2023-09-08 13:42 ` [pve-devel] [RFC pve-cluster 1/6] cluster files: add dhcp.cfg Stefan Hanreich
2023-09-08 13:43 ` [pve-devel] [RFC pve-manager 2/6] sdn: regenerate DHCP config on reload Stefan Hanreich
2023-09-08 13:43 ` [pve-devel] [RFC pve-network 3/6] sdn: dhcp: add abstract class for DHCP plugins Stefan Hanreich
2023-09-08 13:43 ` [pve-devel] [RFC pve-network 4/6] sdn: dhcp: subnet: add DHCP options to subnet configuration Stefan Hanreich
2023-09-11 4:03 ` DERUMIER, Alexandre
2023-09-13 8:37 ` Stefan Hanreich
2023-09-08 13:43 ` [pve-devel] [RFC pve-network 5/6] sdn: dhcp: add DHCP plugin for dnsmasq Stefan Hanreich
2023-09-08 13:43 ` [pve-devel] [RFC pve-network 6/6] sdn: dhcp: regenerate config for DHCP servers on reload Stefan Hanreich
2023-09-11 3:53 ` [pve-devel] [RFC cluster/manager/network 0/6] Add support for DHCP servers to SDN DERUMIER, Alexandre
2023-09-13 8:18 ` DERUMIER, Alexandre
2023-09-13 8:54 ` Stefan Hanreich
2023-09-13 9:26 ` DERUMIER, Alexandre
2023-09-13 11:37 ` Thomas Lamprecht
2023-09-13 11:43 ` DERUMIER, Alexandre
2023-09-13 11:50 ` Stefan Hanreich
2023-09-13 12:40 ` Thomas Lamprecht
2023-09-13 12:50 ` DERUMIER, Alexandre
2023-09-13 13:05 ` Stefan Hanreich
2023-09-13 13:21 ` DERUMIER, Alexandre
2023-09-13 13:48 ` Stefan Hanreich
2023-09-13 13:52 ` Stefan Hanreich
2023-09-14 13:15 ` DERUMIER, Alexandre
2023-09-20 21:48 ` DERUMIER, Alexandre
2023-09-26 11:20 ` Stefan Hanreich
2023-09-26 13:07 ` DERUMIER, Alexandre [this message]
2023-09-26 14:12 ` Stefan Hanreich
2023-09-26 16:55 ` DERUMIER, Alexandre
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cef49dd992e40135c64ffef0e7f9f572784900bc.camel@groupe-cyllene.com \
--to=alexandre.derumier@groupe-cyllene.com \
--cc=pve-devel@lists.proxmox.com \
--cc=s.hanreich@proxmox.com \
--cc=t.lamprecht@proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal