From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E1D47997E7 for ; Wed, 11 Oct 2023 09:55:48 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id AA66D3C78 for ; Wed, 11 Oct 2023 09:55:18 +0200 (CEST) Received: from zimbra.panservice.it (zimbra.panservice.it [212.66.96.48]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 11 Oct 2023 09:55:17 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by zimbra.panservice.it (Postfix) with ESMTP id 3084A2E4B560; Wed, 11 Oct 2023 07:55:17 +0000 (UTC) Received: from zimbra.panservice.it ([127.0.0.1]) by localhost (zimbra.panservice.it [127.0.0.1]) (amavis, port 10032) with ESMTP id Cyv1Hlt17ATJ; Wed, 11 Oct 2023 07:55:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.panservice.it (Postfix) with ESMTP id 1D9102E4B580; Wed, 11 Oct 2023 07:55:13 +0000 (UTC) X-Virus-Scanned: amavis at zimbra.panservice.it Received: from zimbra.panservice.it ([127.0.0.1]) by localhost (zimbra.panservice.it [127.0.0.1]) (amavis, port 10026) with ESMTP id jtcKZy0AVOe4; Wed, 11 Oct 2023 07:55:13 +0000 (UTC) Received: from zimbra.panservice.it (zimbra.panservice.it [212.66.96.48]) by zimbra.panservice.it (Postfix) with ESMTP id 086632E4B57F; Wed, 11 Oct 2023 07:55:13 +0000 (UTC) Date: Wed, 11 Oct 2023 07:55:12 +0000 (UTC) From: Fabrizio Cuseo Reply-To: Fabrizio Cuseo To: Stefan Lendl Cc: Proxmox VE user list , pve-user Message-ID: <652060962.148106.1697010912527.JavaMail.zimbra@zimbra.panservice.it> In-Reply-To: <87jzrtip2a.fsf@gmail.com> References: <1200893807.110513.1696964512431.JavaMail.zimbra@zimbra.panservice.it> <87jzrtip2a.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [212.66.96.161] X-Mailer: Zimbra 8.8.15_GA_4562 (ZimbraWebClient - FF118 (Win)/8.8.15_GA_3) Thread-Topic: qm remote-migrate Thread-Index: hUfFHj22P2Dh53sY2dDhDgADevtn8g== X-SPAM-LEVEL: Spam detection results: 0 AWL 0.846 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] qm remote-migrate X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Oct 2023 07:55:48 -0000 ----- Il 11-ott-23, alle 9:41, Stefan Lendl s.lendl@proxmox.com ha scritto: > Fabrizio Cuseo writes: > > Hello Fabrizio, > > To better understand your issue, the source cluster has a VM with a > bridge with a VLAN tag assigned and the target cluster does not have the > same setup but uses SDN (vnet) without vlan. Yes, it's correct. > After migration you manually changed the VMs configuration to match the > new setup? I can't because remote-migration returns an error (I cannot specify vlan tag on that bridge) > What SDN configuration are you using on the traget cluster? > Please send the output of the following: > > head -n -1 /etc/pve/sdn/*.cfg Can I send you in private ? Because is full of customer's names :/ But, this is a part of files: ==> /etc/pve/sdn/zones.cfg <== vlan: ZonaVLAN bridge vmbr0 ipam pve qinq: VPVT bridge vmbr0 tag 929 ipam pve vlan-protocol 802.1q ==> /etc/pve/sdn/vnets.cfg <== vnet: test100 zone FWHous alias Vlan 100 Test 921 qinq tag 100 vnet: vlan902 zone ZonaVLAN alias Vlan 902 Private-Vlan tag 902 > What was to exact command you ran to start the remote-migrate process? qm remote-migrate 4980 4980 'host=172.16.20.41,apitoken=PVEAPIToken=root@pam!remotemigrate=hiddensecret,fingerprint=hiddenfingerprint' --target-bridge vlan902 --target-storage NfsMirror --online > Did you notice any suspicios log messages in the source clusters > journal? Source: tunnel: -> sending command "version" to remote tunnel: <- got reply 2023-10-10 18:08:48 local WS tunnel version: 2 2023-10-10 18:08:48 remote WS tunnel version: 2 2023-10-10 18:08:48 minimum required WS tunnel version: 2 websocket tunnel started 2023-10-10 18:08:48 starting migration of VM 4980 to node 'nodo01-cluster1' (172.16.20.41) tunnel: -> sending command "bwlimit" to remote tunnel: <- got reply 2023-10-10 18:08:49 found local disk 'CephCluster3Copie:vm-4980-disk-0' (attached) 2023-10-10 18:08:49 mapped: net0 from vmbr1 to vlan902 2023-10-10 18:08:49 Allocating volume for drive 'scsi0' on remote storage 'NfsMirror'.. tunnel: -> sending command "disk" to remote tunnel: <- got reply 2023-10-10 18:08:49 volume 'CephCluster3Copie:vm-4980-disk-0' is 'NfsMirror:4980/vm-4980-disk-0.raw' on the target tunnel: -> sending command "config" to remote tunnel: <- got reply tunnel: -> sending command "start" to remote tunnel: <- got reply 2023-10-10 18:08:50 ERROR: online migrate failure - error - tunnel command '{"start_params":{"forcemachine":"pc-i440fx-8.0+pve0","forcecpu":null,"statefile":"unix","skiplock":1},"cmd":"start","migrate_opts":{"network":null,"nbd ":{"scsi0":{"volid":"NfsMirror:4980/vm-4980-disk-0.raw","success":true,"drivestr":"NfsMirror:4980/vm-4980-disk-0.raw,discard=on,format=raw,size=64G"}},"nbd_proto_version":1,"storagemap":{"default":"NfsMirror"},"migratedfrom":"n ode06-cluster4","type":"websocket","remote_node":"nodo01-cluster1","spice_ticket":null}}' failed - failed to handle 'start' command - start failed: QEMU exited with code 1 2023-10-10 18:08:50 aborting phase 2 - cleanup resources 2023-10-10 18:08:50 migrate_cancel tunnel: -> sending command "stop" to remote tunnel: <- got reply tunnel: -> sending command "quit" to remote tunnel: <- got reply 2023-10-10 18:08:51 ERROR: migration finished with problems (duration 00:00:03) TASK ERROR: migration problems DESTINATION: mtunnel started received command 'version' received command 'bwlimit' received command 'disk' Formatting '/mnt/pve/NfsMirror/images/4980/vm-4980-disk-0.raw', fmt=raw size=68719476736 preallocation=off received command 'config' update VM 4980: -agent 1 -boot order=scsi0;ide2;net0 -cores 2 -ide2 none,media=cdrom -memory 8192 -name SeafileProTestS3 -net0 e1000=86:64:73:AB:33:AE,bridge=vlan902,tag=902 -numa 1 -ostype l26 -scsi0 NfsMirror:4980 /vm-4980-disk-0.raw,discard=on,format=raw,size=64G -scsihw virtio-scsi-pci -smbios1 uuid=39a07e5b-16b5-45a3-aad9-4e3f2b4e87ce -sockets 2 received command 'start' QEMU: vm vlans are not allowed on vnet vlan902 at /usr/share/perl5/PVE/Network/SDN/Zones/Plugin.pm line 228. QEMU: kvm: -netdev type=tap,id=net0,ifname=tap4980i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown: network script /var/lib/qemu-server/pve-bridge failed with status 6400 received command 'stop' received command 'quit' freeing volume 'NfsMirror:4980/vm-4980-disk-0.raw' as part of cleanup disk image '/mnt/pve/NfsMirror/images/4980/vm-4980-disk-0.raw' does not exist switching to exit-mode, waiting for client to disconnect mtunnel exited TASK OK Source VM conf file: agent: 1 balloon: 2048 boot: order=scsi0;ide2;net0 cores: 2 ide2: none,media=cdrom memory: 4096 name: SeafileProTestS3 net0: virtio=86:64:73:AB:33:AE,bridge=vmbr1,tag=902 numa: 1 ostype: l26 scsi0: CephCluster3Copie:vm-4980-disk-0,discard=on,size=64G scsihw: virtio-scsi-pci smbios1: uuid=39a07e5b-16b5-45a3-aad9-4e3f2b4e87ce sockets: 2 vmgenid: 035cd26d-c74e-405e-9b4d-481f26d9cf5f > Usually I would ask you to send me the entire journal but this is not > feasible on the mailing list. If necessary, I would recommend you open a > Thread in our community forum and I will take a look there. > https://forum.proxmox.com/ > > Best regards, > Stefan Lendl Thank you in advance, Fabrizio > >> Hello. >> I am testing qm remote-migrate with 2 pve 8.0.4 clusters. >> Source cluster has one bridge with vlan id on every VM, destination cluster uses >> SDN and a different bridge (vnet) without vlanid. >> If I migrate the vm, i need to specify both bridge and vlan-id, but I have not >> found an option to do it. >> >> PS: after migration, on the new cluster the vm is running without any problem, >> but on source cluster remains locked and in migration, so I need to issue a "qm >> unlock vmid" and stop/delete it. >> >> I know that is an experimental feature, so I send my test results. >> >> Regards, Fabrizio >> >> >> -- >> --- >> Fabrizio Cuseo - mailto:f.cuseo@panservice.it >> >> _______________________________________________ >> pve-user mailing list >> pve-user@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- --- Fabrizio Cuseo - mailto:f.cuseo@panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:info@panservice.it Numero verde nazionale: 800 901492