From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id C1A331FF0B6 for ; Thu, 12 Nov 2026 13:43:26 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 839FC2167B; Thu, 12 Nov 2026 13:42:36 +0100 (CET) Date: Thu, 12 Nov 2026 13:42:18 +0100 From: Arthur Bied-Charreton To: Stefan Hanreich Subject: Re: [RFC pve-network 1/1] sdn: apply changes on all nodes in parallel Message-ID: References: <20260709140625.275618-1-s.hanreich@proxmox.com> <17ce3fca-ab64-426a-b72c-2cb6b12ea4fe@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17ce3fca-ab64-426a-b72c-2cb6b12ea4fe@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794487339923 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.452 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: TQBWNA53QJ7FTJPT2RMER4REDKOH4JKI X-Message-ID-Hash: TQBWNA53QJ7FTJPT2RMER4REDKOH4JKI X-MailFrom: a.bied-charreton@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Fri, Sep 18, 2026 at 01:21:13PM +0200, Stefan Hanreich wrote: > > > On 9/18/26 11:50 AM, Gabriel Goller wrote: > >> + [...] > >> __PACKAGE__->register_method({ > >> name => 'reload', > >> protected => 1, > >> @@ -291,6 +303,7 @@ __PACKAGE__->register_method({ > >> permissions => { > >> check => ['perm', '/sdn', ['SDN.Allocate']], > > > > Hmm this endpoint has SDN.Allocate and the PUT /network endpoint we are calling > > (in pve-manager) needs Sys.Modify. This shouldn't be an issue, but maybe we > > should require Sys.Modify here as well to short-circuit everything, instead of > > making a request to every node. > > Hmm, pre-existing, but what about setups where a user has Sys.Modify only on some nodes? That's > currently possible but probably quite bad, since it allows applying the SDN configuration to > some nodes only. So, I agree - but a breaking change? > afaict the breaking change is already in no? a user with such mixed permissions or only SDN.Alloacte was able to reload all nodes before since it was running with pvesh as root, this commit changes that right? sorry for the noise in case i'm misunderstanding something. > >> }, > >> + expose_credentials => 1, > >> parameters => { > >> additionalProperties => 0, > >> properties => { > >> @@ -336,22 +349,56 @@ __PACKAGE__->register_method({ > >> my $regenerate_frr = ($previous_config_has_frr || $new_config_has_frr) ? 1 : 0; > >> > >> my $code = sub { > >> - $rpcenv->{type} = 'priv'; # to start tasks in background > >> PVE::Cluster::check_cfs_quorum(); > >> + > >> + my $api_client = create_api_client(); > >> my $nodelist = PVE::Cluster::get_nodelist(); > >> + > >> + my $tasks = {}; > >> + > >> for my $node (@$nodelist) { > >> - my $pid = eval { $create_reload_network_worker->($node, $regenerate_frr) }; > >> - warn $@ if $@; > >> + print "$node: reloading network config\n"; > >> + > >> + my $upid = eval { > >> + $api_client->put( > >> + "/nodes/$node/network", > >> + { > >> + 'regenerate-frr' => $regenerate_frr, > >> + }, > >> + ); > >> + }; > >> + > >> + if ($@) { > >> + log_warn("$node: could not reload network configuration: $@\n"); > >> + next; > >> + } > >> + > >> + $tasks->{$upid} = $node; > >> } > >> > >> - # FIXME: use libpve-apiclient (like in cluster join) to create > >> - # tasks and moitor the tasks. > >> + print "waiting for reload tasks to finish\n"; > >> > >> - return; > >> + while ($tasks->%*) { > >> + for my $upid (keys $tasks->%*) { > >> + my $node = $tasks->{$upid}; > >> + my $task = eval { $api_client->get("/nodes/$node/tasks/$upid/status") }; > >> + > >> + if ($@) { > >> + log_warn("could not get status of reload task: $@\n"); > >> + delete $tasks->{$upid}; > >> + next; > >> + } > > > > Hmm should we wait more than 1 second here maybe? What if the reload brings the > > network down shortly... Maybe add a counter and require a task to be gone for > > 3-4 seconds? > > yes, potentially even a re-try logic before giving up completely... > > >> + > >> + next if $task->{status} eq 'running'; > > > > Should we add a exitstatus check here and print a warning like "reload failed" > > or something? > > > >> + print "$node: reload task finished\n"; > >> + delete $tasks->{$upid}; > >> + } > >> + > >> + sleep(1); > > > > Do the pve tasks have a maximum duration where they get killed if they take > > longer? Otherwise a stuck task will make this loop forever. > > > > Although this is a task as well isn't it -- it can just be force-stopped in > > the ui. > > Should be fine imo, due to the possibility of cancelling the task itself. > > >> + } > >> }; > >> > >> return $rpcenv->fork_worker('reloadnetworkall', undef, $authuser, $code); > >> - > >> }, > >> }); > >> > >> -- > >> 2.47.3 > > > > > >