From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id E71301FF187
for <inbox@lore.proxmox.com>; Mon, 26 May 2025 14:20:14 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id BA812311A9;
Mon, 26 May 2025 14:20:15 +0200 (CEST)
Message-ID: <8d2b3e39-23f6-4a7e-b75a-34ca0812baae@proxmox.com>
Date: Mon, 26 May 2025 14:20:09 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
To: pve-devel@lists.proxmox.com
References: <20250526121439.26728-1-h.duerr@proxmox.com>
Content-Language: en-US
From: Hannes Duerr <h.duerr@proxmox.com>
In-Reply-To: <20250526121439.26728-1-h.duerr@proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.932 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
KAM_MAILER 2 Automated Mailer Tag Left in Email
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [ipams.pm, phpipamplugin.pm, networktocode.com, pveplugin.pm,
plugin.pm, nautobotplugin.pm, netboxplugin.pm]
Subject: Re: [pve-devel] [PATCH pve-network v4 1/2] ipam: add Nautobot plugin
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
sorry forgot to increase the version
On 5/26/25 14:14, Hannes Duerr wrote:
> Nautobot is a network source of truth [0] and as such this plugin offers
> the possibility to automatically enter IP addresses and subnets
> (prefixes in Nautobot jargon) into Nautobot as soon as they are created
> in Proxmox VE.
>
> Limitations:
> * The IPAM plugin currently does not recognize whether VMs/CTs are
> active/online and initially sets the status to Active but when the
> VMs/CTs are switched off the status is not deactivated.
> * Nautobot does not support ranges that can be searched for free ip
> addresses, so we map as many pool prefixes that we search until we
> have searched the entire range for free ip addresses
> * Nautobot has the possibility to map DNS names to IP addresses.
> However, since we have no standardized way to set the DNS names in
> Proxmox VE (for containers the container name in Proxmox VE is set as
> the host name in the container, but this is not possible for VMs) we
> refrain from setting a DNS name at the moment.
> * Nautobot does not have a field in IP address objects for a MAC address
> or to mark them as gateway, so we write this in the comment.
>
> [0] https://networktocode.com/nautobot/
>
> Co-authored-by: lou lecrivain <lou.lecrivain@wdz.de>
> Signed-off-by: Hannes Duerr <h.duerr@proxmox.com>
> ---
>
> Notes:
> Changes from v4 -> v5:
> * instead of skipping already created prefixes, search them
>
> Changes from v3 -> v4:
> * Merge the check for empty prefixes/subnets into this commit
> * Create nautobot_api_request to unify API calls
> * add update_subnet function
> * fix add_range_next_freeip
> * fix and unify error handling
> * Fix perl style
>
> src/PVE/API2/Network/SDN/Ipams.pm | 1 +
> src/PVE/Network/SDN/Ipams.pm | 3 +
> src/PVE/Network/SDN/Ipams/Makefile | 2 +-
> src/PVE/Network/SDN/Ipams/NautobotPlugin.pm | 510 ++++++++++++++++++++
> 4 files changed, 515 insertions(+), 1 deletion(-)
> create mode 100644 src/PVE/Network/SDN/Ipams/NautobotPlugin.pm
>
> diff --git a/src/PVE/API2/Network/SDN/Ipams.pm b/src/PVE/API2/Network/SDN/Ipams.pm
> index 27ead02..8074512 100644
> --- a/src/PVE/API2/Network/SDN/Ipams.pm
> +++ b/src/PVE/API2/Network/SDN/Ipams.pm
> @@ -12,6 +12,7 @@ use PVE::Network::SDN::Ipams::Plugin;
> use PVE::Network::SDN::Ipams::PVEPlugin;
> use PVE::Network::SDN::Ipams::PhpIpamPlugin;
> use PVE::Network::SDN::Ipams::NetboxPlugin;
> +use PVE::Network::SDN::Ipams::NautobotPlugin;
> use PVE::Network::SDN::Dhcp;
> use PVE::Network::SDN::Vnets;
> use PVE::Network::SDN::Zones;
> diff --git a/src/PVE/Network/SDN/Ipams.pm b/src/PVE/Network/SDN/Ipams.pm
> index c689b8f..2ecb75e 100644
> --- a/src/PVE/Network/SDN/Ipams.pm
> +++ b/src/PVE/Network/SDN/Ipams.pm
> @@ -12,11 +12,14 @@ use PVE::Network;
>
> use PVE::Network::SDN::Ipams::PVEPlugin;
> use PVE::Network::SDN::Ipams::NetboxPlugin;
> +use PVE::Network::SDN::Ipams::NautobotPlugin;
> use PVE::Network::SDN::Ipams::PhpIpamPlugin;
> use PVE::Network::SDN::Ipams::Plugin;
>
> +
> PVE::Network::SDN::Ipams::PVEPlugin->register();
> PVE::Network::SDN::Ipams::NetboxPlugin->register();
> +PVE::Network::SDN::Ipams::NautobotPlugin->register();
> PVE::Network::SDN::Ipams::PhpIpamPlugin->register();
> PVE::Network::SDN::Ipams::Plugin->init();
>
> diff --git a/src/PVE/Network/SDN/Ipams/Makefile b/src/PVE/Network/SDN/Ipams/Makefile
> index 4e7d65f..75e5b9a 100644
> --- a/src/PVE/Network/SDN/Ipams/Makefile
> +++ b/src/PVE/Network/SDN/Ipams/Makefile
> @@ -1,4 +1,4 @@
> -SOURCES=Plugin.pm PhpIpamPlugin.pm NetboxPlugin.pm PVEPlugin.pm
> +SOURCES=Plugin.pm PhpIpamPlugin.pm NetboxPlugin.pm PVEPlugin.pm NautobotPlugin.pm
>
>
> PERL5DIR=${DESTDIR}/usr/share/perl5
> diff --git a/src/PVE/Network/SDN/Ipams/NautobotPlugin.pm b/src/PVE/Network/SDN/Ipams/NautobotPlugin.pm
> new file mode 100644
> index 0000000..a6eb7ff
> --- /dev/null
> +++ b/src/PVE/Network/SDN/Ipams/NautobotPlugin.pm
> @@ -0,0 +1,510 @@
> +package PVE::Network::SDN::Ipams::NautobotPlugin;
> +
> +use strict;
> +use warnings;
> +use PVE::INotify;
> +use PVE::Cluster;
> +use PVE::Tools;
> +use NetAddr::IP;
> +use Net::Subnet qw(subnet_matcher);
> +
> +use base('PVE::Network::SDN::Ipams::Plugin');
> +
> +sub type {
> + return 'nautobot';
> +}
> +
> +sub properties {
> + return {
> + namespace => {
> + type => 'string',
> + },
> + };
> +}
> +
> +sub options {
> + return {
> + url => { optional => 0 },
> + token => { optional => 0 },
> + namespace => { optional => 0 },
> + fingerprint => { optional => 1 },
> + };
> +}
> +
> +sub default_ip_status {
> + return 'Active';
> +}
> +
> +sub nautobot_api_request {
> + my ($config, $method, $path, $params) = @_;
> +
> + return PVE::Network::SDN::api_request(
> + $method,
> + "$config->{url}${path}",
> + [
> + 'Content-Type' => 'application/json; charset=UTF-8',
> + 'Authorization' => "token $config->{token}",
> + 'Accept' => "application/json",
> + ],
> + $params,
> + $config->{fingerprint},
> + );
> +}
> +
> +sub add_subnet {
> + my ($class, $config, undef, $subnet, $noerr) = @_;
> +
> + my $cidr = $subnet->{cidr};
> + my $namespace = $config->{namespace};
> +
> + my $internalid = get_prefix_id($config, $cidr, $noerr);
> + if ($internalid) {
> + return if $noerr;
> + die "could not add the subnet $subnet because it already exists in nautobot\n";
> + }
> +
> + my $params = {
> + prefix => $cidr,
> + namespace => $namespace,
> + status => default_ip_status(),
> + };
> +
> + eval { nautobot_api_request($config, "POST", "/ipam/prefixes/", $params); };
> + if ($@) {
> + return if $noerr;
> + die "error adding the subnet $subnet to nautobot $@\n";
> + }
> +}
> +
> +sub update_subnet {
> + my ($class, $plugin_config, $subnetid, $subnet, $old_subnet, $noerr) = @_;
> + # dhcp ranges are not supported in nautobot so we don't have to update them
> +}
> +
> +sub del_subnet {
> + my ($class, $config, $subnetid, $subnet, $noerr) = @_;
> +
> + my $cidr = $subnet->{cidr};
> +
> + my $internalid = get_prefix_id($config, $cidr, $noerr);
> + if (!$internalid) {
> + warn("could not find delete the subnet $cidr because it does not exist in nautobot\n");
> + return;
> + }
> +
> + if (!subnet_is_deletable($config, $subnetid, $subnet, $internalid, $noerr)) {
> + return if $noerr;
> + die "could not delete the subnet $cidr, it still contains ip addresses!\n";
> + }
> +
> + # delete associated gateway IP addresses
> + $class->empty_subnet($config, $subnetid, $subnet, $internalid, $noerr);
> +
> + eval { nautobot_api_request($config, "DELETE", "/ipam/prefixes/$internalid/"); };
> + if ($@) {
> + return if $noerr;
> + die "error deleting subnet from nautobot: $@\n";
> + }
> + return 1;
> +}
> +
> +sub add_ip {
> + my ($class, $config, undef, $subnet, $ip, $hostname, $mac, undef, $is_gateway, $noerr) = @_;
> +
> + my $mask = $subnet->{mask};
> + my $namespace = $config->{namespace};
> +
> + my $description = undef;
> + if ($is_gateway) {
> + $description = 'gateway';
> + } elsif ($mac) {
> + $description = "mac:$mac";
> + }
> +
> + my $params = {
> + address => "$ip/$mask",
> + type => "dhcp",
> + description => $description,
> + namespace => $namespace,
> + status => default_ip_status(),
> + };
> +
> + eval { nautobot_api_request($config, "POST", "/ipam/ip-addresses/", $params); };
> +
> + if ($@) {
> + if ($is_gateway) {
> + die "error add subnet ip to ipam: ip $ip already exist: $@"
> + if !is_ip_gateway($config, $ip, $noerr);
> + } elsif (!$noerr) {
> + die "error add subnet ip to ipam: ip already exist: $@";
> + }
> + }
> +}
> +
> +sub add_next_freeip {
> + my ($class, $config, undef, $subnet, $hostname, $mac, undef, $noerr) = @_;
> +
> + my $cidr = $subnet->{cidr};
> + my $namespace = $config->{namespace};
> +
> + my $internalid = get_prefix_id($config, $cidr, $noerr);
> + if (!defined($internalid)) {
> + return if $noerr;
> + die "could not find prefix $cidr in nautobot\n";
> + }
> +
> + my $description = undef;
> + $description = "mac:$mac" if $mac;
> +
> + my $params = {
> + type => "dhcp",
> + description => $description,
> + namespace => $namespace,
> + status => default_ip_status(),
> + };
> +
> + my $response = eval {
> + return nautobot_api_request(
> + $config, "POST", "/ipam/prefixes/$internalid/available-ips/", $params,
> + );
> + };
> + if ($@ || !$response) {
> + return if $noerr;
> + die "could not allocate ip in subnet $cidr: $@\n";
> + }
> +
> + my $ip = NetAddr::IP->new($response->{address});
> +
> + return $ip->addr;
> +}
> +
> +sub find_ip_in_prefix {
> + my ($config, $prefix_id, $limit, $start_range, $end_range) = @_;
> +
> + # Fetch available IPs from the temporary pool and find a matching IP
> + my $result = eval {
> + return nautobot_api_request(
> + $config,
> + "GET",
> + "/ipam/prefixes/$prefix_id/available-ips/?limit=$limit",
> + );
> + };
> +
> + # search list for IPs in actual range
> + if (!$@ && defined($result)) {
> + foreach my $entry (@$result) {
> + my $ip = NetAddr::IP->new($entry->{address});
> + # comparison is only possible because they are in the same subnet
> + if ($start_range <= $ip && $ip <= $end_range) {
> + return $ip->addr;
> + }
> + }
> + }
> + return;
> +}
> +
> +sub add_range_next_freeip {
> + my ($class, $config, $subnet, $range, $data, $noerr) = @_;
> +
> + my $cidr = NetAddr::IP->new($subnet->{cidr});
> + my $namespace = $config->{namespace};
> +
> + # Nautobot does not support IP ranges, only prefixes.
> + # Therefore we divide the range into smaller pool prefixes,
> + # each containing 256 addresses, and search them for available IPs
> + my $prefix_size = $cidr->version == 4 ? 24 : 120;
> + my $increment = 256;
> + my $found_ip = undef;
> +
> + my $start_range = NetAddr::IP->new($range->{'start-address'}, $prefix_size);
> + my $end_range = NetAddr::IP->new($range->{'end-address'}, $prefix_size);
> + my $matcher = subnet_matcher($end_range->cidr);
> + my $current_ip = $start_range;
> +
> + while (1) {
> + my $current_cidr = $current_ip->addr . "/$prefix_size";
> +
> + my $params = {
> + prefix => $current_cidr,
> + namespace => $namespace,
> + status => default_ip_status(),
> + type => "pool",
> + };
> +
> + my $prefix_id = get_prefix_id($config, $current_cidr, $noerr);
> + if ($prefix_id) {
> + # search the existing prefix for valid ip
> + $found_ip =
> + find_ip_in_prefix($config, $prefix_id, $increment, $start_range, $end_range);
> + } else {
> + # create temporary pool prefix
> + my $temp_prefix =
> + eval { return nautobot_api_request($config, "POST", "/ipam/prefixes/", $params); };
> +
> + my $temp_prefix_id = $temp_prefix->{id};
> +
> + # search temporarly created prefix
> + $found_ip =
> + find_ip_in_prefix($config, $temp_prefix_id, $increment, $start_range, $end_range);
> +
> + # Delete temporary prefix pool
> + eval { nautobot_api_request($config, "DELETE", "/ipam/prefixes/$temp_prefix_id/"); };
> + }
> +
> + last if $found_ip;
> +
> + # we searched the last pool prefix
> + last if $matcher->($current_ip->addr);
> +
> + $current_ip = $current_ip->plus($increment);
> + }
> +
> + if (!$found_ip) {
> + return if $noerr;
> + die "could not allocate ip in the range "
> + . $start_range->addr . " - "
> + . $end_range->addr
> + . ": $@\n";
> + }
> +
> + $class->add_ip(
> + $config,
> + undef,
> + $subnet,
> + $found_ip,
> + $data->{hostname},
> + $data->{mac},
> + undef,
> + 0,
> + $noerr,
> + );
> +
> + return $found_ip;
> +}
> +
> +sub update_ip {
> + my ($class, $config, $subnetid, $subnet, $ip, $hostname, $mac, undef, $is_gateway, $noerr) = @_;
> +
> + my $mask = $subnet->{mask};
> + my $namespace = $config->{namespace};
> +
> + my $description = undef;
> + if ($is_gateway) {
> + $description = 'gateway';
> + } elsif ($mac) {
> + $description = "mac:$mac";
> + }
> +
> + my $params = {
> + address => "$ip/$mask",
> + type => "dhcp",
> + description => $description,
> + namespace => $namespace,
> + status => default_ip_status(),
> + };
> +
> + my $ip_id = get_ip_id($config, $ip, $noerr);
> + if (!defined($ip_id)) {
> + return if $noerr;
> + die "could not find the ip $ip in nautobot\n";
> + }
> +
> + eval { nautobot_api_request($config, "PATCH", "/ipam/ip-addresses/$ip_id/", $params); };
> + if ($@) {
> + return if $noerr;
> + die "error updating ip $ip: $@";
> + }
> +}
> +
> +sub del_ip {
> + my ($class, $config, undef, undef, $ip, $noerr) = @_;
> +
> + return if !$ip;
> +
> + my $ip_id = get_ip_id($config, $ip, $noerr);
> + if (!defined($ip_id)) {
> + warn("could not find the ip $ip in nautobot\n");
> + return;
> + }
> +
> + eval { nautobot_api_request($config, "DELETE", "/ipam/ip-addresses/$ip_id/"); };
> + if ($@) {
> + return if $noerr;
> + die "error deleting ip $ip : $@\n";
> + }
> +
> + return 1;
> +}
> +
> +sub empty_subnet {
> + my ($class, $config, $subnetid, $subnet, $subnetuuid, $noerr) = @_;
> +
> + my $namespace = $config->{namespace};
> +
> + my $response = eval {
> + return nautobot_api_request(
> + $config,
> + "GET",
> + "/ipam/ip-addresses/?namespace=$namespace&parent=$subnetuuid",
> + );
> + };
> + if ($@) {
> + return if $noerr;
> + die "could not find the subnet $subnet in nautobot: $@\n";
> + }
> +
> + for my $ip (@{ $response->{results} }) {
> + del_ip($class, $config, undef, undef, $ip->{host}, $noerr);
> + }
> +
> + return 1;
> +}
> +
> +sub subnet_is_deletable {
> + my ($config, $subnetid, $subnet, $subnetuuid, $noerr) = @_;
> +
> + my $namespace = $config->{namespace};
> +
> + my $response = eval {
> + return nautobot_api_request(
> + $config,
> + "GET",
> + "/ipam/ip-addresses/?namespace=$namespace&parent=$subnetuuid",
> + );
> + };
> + if ($@) {
> + return if $noerr;
> + die "error querying prefix $subnet: $@\n";
> + }
> + my $n_ips = scalar $response->{results}->@*;
> +
> + # least costly check operation 1st
> + return 1 if ($n_ips == 0);
> +
> + for my $ip (values $response->{results}->@*) {
> + if (!is_ip_gateway($config, $ip->{host}, $noerr)) {
> + # some remaining IP is not a gateway so we can't delete the subnet
> + return 0;
> + }
> + }
> + #all remaining IPs are gateways
> + return 1;
> +}
> +
> +sub verify_api {
> + my ($class, $config) = @_;
> +
> + my $namespace = $config->{namespace};
> +
> + # check if the namespace and the status "Active" exist
> + eval {
> + get_namespace_id($config, $namespace) // die "namespace $namespace does not exist";
> + get_status_id($config, default_ip_status())
> + // die "the status " . default_ip_status() . " does not exist";
> + };
> + if ($@) {
> + die "could not use nautobot api: $@\n";
> + }
> +}
> +
> +sub get_ips_from_mac {
> + my ($class, $config, $mac, $zone) = @_;
> +
> + my $ip4 = undef;
> + my $ip6 = undef;
> +
> + my $data = eval { nautobot_api_request($config, "GET", "/ipam/ip-addresses/?q=$mac"); };
> + if ($@) {
> + die "could not query ip address entry for mac $mac: $@";
> + }
> +
> + for my $ip (@{ $data->{results} }) {
> + if ($ip->{ip_version} == 4 && !$ip4) {
> + ($ip4, undef) = split(/\//, $ip->{address});
> + }
> +
> + if ($ip->{ip_version} == 6 && !$ip6) {
> + ($ip6, undef) = split(/\//, $ip->{address});
> + }
> + }
> +
> + return ($ip4, $ip6);
> +}
> +
> +sub on_update_hook {
> + my ($class, $config) = @_;
> +
> + PVE::Network::SDN::Ipams::NautobotPlugin::verify_api($class, $config);
> +}
> +
> +sub get_ip_id {
> + my ($config, $ip, $noerr) = @_;
> +
> + my $result =
> + eval { return nautobot_api_request($config, "GET", "/ipam/ip-addresses/?address=$ip"); };
> + if ($@) {
> + return if $noerr;
> + die "error while querying for ip $ip id: $@\n";
> + }
> +
> + my $data = @{ $result->{results} }[0];
> + return $data->{id};
> +}
> +
> +sub get_prefix_id {
> + my ($config, $cidr, $noerr) = @_;
> +
> + my $result =
> + eval { return nautobot_api_request($config, "GET", "/ipam/prefixes/?prefix=$cidr"); };
> + if ($@) {
> + return if $noerr;
> + die "error while querying for cidr $cidr prefix id: $@\n";
> + }
> +
> + my $data = @{ $result->{results} }[0];
> + return $data->{id};
> +}
> +
> +sub get_namespace_id {
> + my ($config, $namespace, $noerr) = @_;
> +
> + my $result =
> + eval { return nautobot_api_request($config, "GET", "/ipam/namespaces/?name=$namespace"); };
> + if ($@) {
> + return if $noerr;
> + die "error while querying for namespace $namespace id: $@\n";
> + }
> +
> + my $data = @{ $result->{results} }[0];
> + return $data->{id};
> +}
> +
> +sub get_status_id {
> + my ($config, $status, $noerr) = @_;
> +
> + my $result =
> + eval { return nautobot_api_request($config, "GET", "/extras/statuses/?name=$status"); };
> + if ($@) {
> + return if $noerr;
> + die "error while querying for status $status id: $@\n";
> + }
> +
> + my $data = @{ $result->{results} }[0];
> + return $data->{id};
> +}
> +
> +sub is_ip_gateway {
> + my ($config, $ip, $noerr) = @_;
> +
> + my $result =
> + eval { return nautobot_api_request($config, "GET", "/ipam/ip-addresses/?address=$ip"); };
> + if ($@) {
> + return if $noerr;
> + die "error while checking if $ip is a gateway: $@\n";
> + }
> +
> + my $data = @{ $result->{results} }[0];
> + return $data->{description} eq 'gateway';
> +}
> +
> +1;
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel