From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 1074A1FF09B for ; Mon, 12 Oct 2026 12:00:06 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 39C502139F; Mon, 12 Oct 2026 12:00:04 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Mon, 12 Oct 2026 11:59:56 +0200 Message-Id: From: "Daniel Kral" To: "Fiona Ebner" , Subject: Re: [PATCH ha-manager v3 10/21] api: resources: migration: support additional migration options X-Mailer: aerc 0.22.0-10-g6373ac9d2179-dirty References: <20260918160841.128088-1-f.ebner@proxmox.com> <20260918160841.128088-11-f.ebner@proxmox.com> In-Reply-To: <20260918160841.128088-11-f.ebner@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1791799196294 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.842 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 4TCT3F4RNVH7QQJT7MCIPRNNZQ5SWJDW X-Message-ID-Hash: 4TCT3F4RNVH7QQJT7MCIPRNNZQ5SWJDW X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Fri Sep 18, 2026 at 6:08 PM CEST, Fiona Ebner wrote: > Additional migration options can now be passed from the guest API > endpoints to the HA stack. From the HA resource API endpoint, the > options are passed all the way to the resource plugins. > > Check that all nodes are recent enough to support the new-style JSON > CRM command, otherwise produce a warning and fall-back to the previous > behavior ignoring the additional migration options. > > The migration API endpoints for containers and VMs use different > parameters. For example, the endpoint for containers does not have > the 'with-conntrack-state' option. To keep the schema for the more > abstract endpoint for HA resources clean, separate by type using a > oneOf schema and let the resource plugins declare additional > parameters via a migrate_json_properties() function. > > Initially, this will be used to fix #7053 and pass along the > 'with-conntrack-state' migration option. > > Signed-off-by: Fiona Ebner > --- > > The node version might need to be adapted when applying! > > New in v3. > > src/PVE/API2/HA/Resources.pm | 99 ++++++++++++++++++++++++++++++----- > src/PVE/HA/Resources/PVECT.pm | 10 ++++ > src/PVE/HA/Resources/PVEVM.pm | 10 ++++ > 3 files changed, 105 insertions(+), 14 deletions(-) > > diff --git a/src/PVE/API2/HA/Resources.pm b/src/PVE/API2/HA/Resources.pm > index b96572b..6d808ca 100644 > --- a/src/PVE/API2/HA/Resources.pm > +++ b/src/PVE/API2/HA/Resources.pm > @@ -3,6 +3,8 @@ package PVE::API2::HA::Resources; > use strict; > use warnings; > =20 > +use JSON qw(); > + > use PVE::SafeSyslog; > use PVE::Tools qw(extract_param); > use PVE::Cluster; > @@ -12,6 +14,7 @@ use HTTP::Status qw(:constants); > use Storable qw(dclone); > use PVE::JSONSchema qw(get_standard_option); > use PVE::RPCEnvironment; > +use PVE::SafeSyslog; > =20 > use PVE::RESTHandler; > =20 > @@ -34,6 +37,34 @@ my $api_copy_config =3D sub { > return $scfg; > }; > =20 > +my $common_migrate_json_properties =3D { > + sid =3D> get_standard_option( > + 'pve-ha-resource-or-vm-id', > + { completion =3D> \&PVE::HA::Tools::complete_sid }, > + ), > + node =3D> get_standard_option( > + 'pve-node', > + { > + completion =3D> \&PVE::Cluster::complete_migration_target, > + description =3D> "Target node.", > + }, > + ), > +}; > + > +my sub json_crm_command_is_supported { nit: This assumes that cfs_update() is called before, which is called in PVE::RPCEnvironment::init_request() for API requests and PVE::RESTEnvironment::setup_default_cli_env() for CLI requests, that is used in PVE::CLI::ha_manager::setup_environment(). Maybe we should add a comment about that, because otherwise the 'version-info' KV will only contain the current node's JSON string, therefore checking only locally? Could be pedantic though. > + my $version_info =3D PVE::Cluster::get_node_kv('version-info'); > + for my $node (sort keys $version_info->%*) { > + my $node_info =3D eval { JSON::decode_json($version_info->{$node= }); }; > + if (my $err =3D $@) { # warn, but continue > + syslog('warn', "cannot parse version info for $node as JSON = - $err"); Shouldn't we fail if it's not possible to determine a node's version? Any way, the syslog should probably have some context why parsing the version info is needed here and should also be shown to stdout/stderr for callers of ha-manager. > + } elsif (!PVE::Cluster::pvecfg_min_version($node_info->{version}= , 9, 2, 21)) { > + return 0; > + } > + } > + > + return 1; > +} IMO a generic version of this helper would be nice to have in PVE::Cluster as well, as there already are users that need that functionality (David's vNet pool series [0] and Lukas' notification matcher series [1]), but no blocker for this. [0] https://lore.proxmox.com/pve-devel/20260724142528.109453-4-d.riley@prox= mox.com/ [1] https://lore.proxmox.com/pve-devel/20261009125646.284673-39-l.wagner@pr= oxmox.com/ > + > sub check_service_state { > my ($sid, $req_state) =3D @_; > =20 > @@ -336,20 +367,34 @@ __PACKAGE__->register_method({ > check =3D> ['perm', '/', ['Sys.Console']], > }, > parameters =3D> { > - additionalProperties =3D> 0, > - properties =3D> { > - sid =3D> get_standard_option( > - 'pve-ha-resource-or-vm-id', > - { completion =3D> \&PVE::HA::Tools::complete_sid }, > - ), > - node =3D> get_standard_option( > - 'pve-node', > - { > - completion =3D> \&PVE::Cluster::complete_migration_t= arget, > - description =3D> "Target node.", > - }, > - ), > + 'type-property' =3D> 'resource-type', > + 'type-property-schema' =3D> { > + type =3D> 'string', > + description =3D> 'The resource type. Automatically determine= d from the service ID.', > + enum =3D> ['ct', 'vm'], > }, > + oneOf =3D> [ > + { > + 'instance-type' =3D> 'ct', > + additionalProperties =3D> 0, > + properties =3D> PVE::HA::Resources::PVECT::migrate_json_= properties( > + $common_migrate_json_properties), > + }, > + { > + 'instance-type' =3D> 'vm', > + additionalProperties =3D> 0, > + properties =3D> PVE::HA::Resources::PVEVM::migrate_json_= properties( > + $common_migrate_json_properties), > + }, > + ], > + }, > + resolve_type =3D> sub { > + my ($param) =3D @_; > + my $type =3D (PVE::HA::Config::parse_sid($param->{sid}))[1]; > + if (!($type eq 'ct' || $type eq 'vm')) { > + die "Unknown service type '$type' for 'migrate' endpoint sch= ema.\n"; > + } > + return $type; > }, > returns =3D> { > type =3D> 'object', > @@ -403,12 +448,38 @@ __PACKAGE__->register_method({ > =20 > my ($sid, $type, $name) =3D PVE::HA::Config::parse_sid(extract_p= aram($param, 'sid')); > my $req_node =3D extract_param($param, 'node'); > + my $resource_type =3D extract_param($param, 'resource-type'); > + if ($resource_type ne $type) { > + raise_param_exc({ > + 'resource-type' =3D> "$resource_type does not match type= of $sid" }); > + } > + > + # The rest of the parameters in $param are migration options pas= sed along to the plugin. > =20 > PVE::HA::Config::service_is_ha_managed($sid); > =20 > check_service_state($sid); > =20 > - PVE::HA::Config::queue_crm_commands("migrate $sid $req_node"); > + my $crm_command; > + if (json_crm_command_is_supported()) { > + $crm_command =3D JSON::encode_json({ > + kind =3D> 'migrate', > + sid =3D> $sid, > + node =3D> $req_node, > + options =3D> $param, > + }); > + } else { > + if (scalar(keys $param->%*) > 0) { > + syslog( > + 'warn', > + "$sid migration to $req_node: ignoring additional mi= gration options" > + . " - not supported by CRM on all nodes", > + ); nit: This should also be shown in stdout/stderr to direct callers of ha-manager. > + } > + $crm_command =3D "migrate $sid $req_node"; > + } > + > + PVE::HA::Config::queue_crm_commands($crm_command); Nice! > $result->{sid} =3D $sid; > $result->{'requested-node'} =3D $req_node; > =20 > diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.p= m > index 3d8a5b9..17dd697 100644 > --- a/src/PVE/HA/Resources/PVECT.pm > +++ b/src/PVE/HA/Resources/PVECT.pm > @@ -174,4 +174,14 @@ sub get_static_stats_from_config { > }; > } > =20 > +# FIXME: the extra properties are missing from the docs, since the PVE::= API::LXC module can't be > +# included in that environment, because of cyclic dependencies.. > +sub migrate_json_properties { > + my ($props) =3D @_; > + my $extra_migrate_props =3D eval { PVE::API2::LXC::ha_migrate_json_p= roperties() } // {}; > + die $@ if $@ && !$ENV{PVE_GENERATING_DOCS}; > + $props->{$_} =3D $extra_migrate_props->{$_} for keys $extra_migrate_= props->%*; This will write the specific migrate props in the external $props, i.e., $common_migrate_json_properties. For some reason, the one for VMs is run first for me, which results in both VMs and CTs having --with-conntrack-state in the end: # ha-manager help migrate USAGE: ha-manager crm-command migrate --resource-type = [OPTIONS] Request resource migration (online) to another node. : HA resource ID. This consists of a resource type followed by a resource specific name, separated with colon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM or CT id as a shortcut (example: 100). Target node. --resource-type The resource type. Automatically determined from the service ID. Conditional options: [resource-type=3Dct] --skip-config-check (default=3D0) For intra-cluster migration. Skip checking if the configuration can be parsed successfully by the target. Using this flag can lead to dropped configuration options if the target is too old to parse them. --with-conntrack-state (default=3D0) Whether to migrate conntrack entries for running VMs. [resource-type=3Dvm] --skip-config-check (default=3D0) For intra-cluster migration. Skip checking if the configuration can be parsed successfully by the target. Using this flag can lead to dropped configuration options if the target is too old to parse them. --with-conntrack-state (default=3D0) Whether to migrate conntrack entries for running VMs. Therefore, this should only write in the new copy made by ha_migrate_json_properties(). > + return $props; > +} > + > 1; > diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.p= m > index 964f374..52a48e4 100644 > --- a/src/PVE/HA/Resources/PVEVM.pm > +++ b/src/PVE/HA/Resources/PVEVM.pm > @@ -194,4 +194,14 @@ sub get_static_stats_from_config { > }; > } > =20 > +# FIXME: the extra properties are missing from the docs, since the PVE::= API::QEMU module can't be > +# included in that environment, because of cyclic dependencies.. > +sub migrate_json_properties { > + my ($props) =3D @_; > + my $extra_migrate_props =3D eval { PVE::API2::Qemu::ha_migrate_json_= properties() } // {}; > + die $@ if $@ && !$ENV{PVE_GENERATING_DOCS}; > + $props->{$_} =3D $extra_migrate_props->{$_} for keys $extra_migrate_= props->%*; > + return $props; > +} Same here > + > 1;