From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2150E99C0C for ; Wed, 11 Oct 2023 12:14:30 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 049C372F8 for ; Wed, 11 Oct 2023 12:14:30 +0200 (CEST) Received: from zimbra.panservice.it (zimbra.panservice.it [212.66.96.48]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 11 Oct 2023 12:14:28 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by zimbra.panservice.it (Postfix) with ESMTP id 7C7402E4BBCC; Wed, 11 Oct 2023 10:14:28 +0000 (UTC) Received: from zimbra.panservice.it ([127.0.0.1]) by localhost (zimbra.panservice.it [127.0.0.1]) (amavis, port 10032) with ESMTP id 22VSxRyndDO2; Wed, 11 Oct 2023 10:14:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.panservice.it (Postfix) with ESMTP id E6F2C2E4BBCD; Wed, 11 Oct 2023 10:14:24 +0000 (UTC) X-Virus-Scanned: amavis at zimbra.panservice.it Received: from zimbra.panservice.it ([127.0.0.1]) by localhost (zimbra.panservice.it [127.0.0.1]) (amavis, port 10026) with ESMTP id SXTKXoWLEHf5; Wed, 11 Oct 2023 10:14:24 +0000 (UTC) Received: from zimbra.panservice.it (zimbra.panservice.it [212.66.96.48]) by zimbra.panservice.it (Postfix) with ESMTP id D32CF2E4BBCC; Wed, 11 Oct 2023 10:14:24 +0000 (UTC) Date: Wed, 11 Oct 2023 10:14:24 +0000 (UTC) From: Fabrizio Cuseo Reply-To: Fabrizio Cuseo To: Stefan Lendl Cc: pve-user Message-ID: <818088054.154408.1697019264217.JavaMail.zimbra@zimbra.panservice.it> In-Reply-To: <87zg0plcss.fsf@gmail.com> References: <1200893807.110513.1696964512431.JavaMail.zimbra@zimbra.panservice.it> <87jzrtip2a.fsf@gmail.com> <652060962.148106.1697010912527.JavaMail.zimbra@zimbra.panservice.it> <87zg0plcss.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [212.66.96.161] X-Mailer: Zimbra 8.8.15_GA_4562 (ZimbraWebClient - FF118 (Win)/8.8.15_GA_3) Thread-Topic: qm remote-migrate Thread-Index: zb2MYmIjuSleRHCBsmQUyQwejUcb2A== X-SPAM-LEVEL: Spam detection results: 0 AWL 0.818 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] qm remote-migrate X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Oct 2023 10:14:30 -0000 ----- Il 11-ott-23, alle 11:37, Stefan Lendl s.lendl@proxmox.com ha scritto: > Fabrizio Cuseo writes: > > Thanks for providing the details. > I will investigate the situation and we will consider a solution for our > upcoming SDN upgrade. > > As a solution for now, please try to remove the VLAN tag from the source VM > and try to migrate again. The target net interface will not require a > VLAN tag assigned (and is therefore not allowed) on the VM because it is > configured via SDN already. Yes, I have done with a test VM, but I can't do it with production VMs because if I remove vlan tag, the source VM will stop to work. But I can install and configure SDN on the source cluster (upgrading it to the last 8.x), create a vlan zone, a vnet with that vlan id, change source bridge to vnet bridge and removing the vlan tag, and migrate. (I have just tested and seems to work). Thank you again, Fabrizio > > Best regards, > Stefan > >> ----- Il 11-ott-23, alle 9:41, Stefan Lendl s.lendl@proxmox.com ha scritto: >> >>> Fabrizio Cuseo writes: >>> >>> Hello Fabrizio, >>> >>> To better understand your issue, the source cluster has a VM with a >>> bridge with a VLAN tag assigned and the target cluster does not have the >>> same setup but uses SDN (vnet) without vlan. >> >> Yes, it's correct. >> >> >>> After migration you manually changed the VMs configuration to match the >>> new setup? >> >> I can't because remote-migration returns an error (I cannot specify vlan tag on >> that bridge) >> >>> What SDN configuration are you using on the traget cluster? >>> Please send the output of the following: >>> >>> head -n -1 /etc/pve/sdn/*.cfg >> >> Can I send you in private ? Because is full of customer's names :/ >> >> But, this is a part of files: >> >> ==> /etc/pve/sdn/zones.cfg <== >> vlan: ZonaVLAN >> bridge vmbr0 >> ipam pve >> >> qinq: VPVT >> bridge vmbr0 >> tag 929 >> ipam pve >> vlan-protocol 802.1q >> >> >> ==> /etc/pve/sdn/vnets.cfg <== >> vnet: test100 >> zone FWHous >> alias Vlan 100 Test 921 qinq >> tag 100 >> >> vnet: vlan902 >> zone ZonaVLAN >> alias Vlan 902 Private-Vlan >> tag 902 >> >> >> >> >>> What was to exact command you ran to start the remote-migrate process? >> >> qm remote-migrate 4980 4980 >> 'host=172.16.20.41,apitoken=PVEAPIToken=root@pam!remotemigrate=hiddensecret,fingerprint=hiddenfingerprint' >> --target-bridge vlan902 --target-storage NfsMirror --online >> >> >> >>> Did you notice any suspicios log messages in the source clusters >>> journal? >> >> Source: >> >> tunnel: -> sending command "version" to remote >> tunnel: <- got reply >> 2023-10-10 18:08:48 local WS tunnel version: 2 >> 2023-10-10 18:08:48 remote WS tunnel version: 2 >> 2023-10-10 18:08:48 minimum required WS tunnel version: 2 >> websocket tunnel started >> 2023-10-10 18:08:48 starting migration of VM 4980 to node 'nodo01-cluster1' >> (172.16.20.41) >> tunnel: -> sending command "bwlimit" to remote >> tunnel: <- got reply >> 2023-10-10 18:08:49 found local disk 'CephCluster3Copie:vm-4980-disk-0' >> (attached) >> 2023-10-10 18:08:49 mapped: net0 from vmbr1 to vlan902 >> 2023-10-10 18:08:49 Allocating volume for drive 'scsi0' on remote storage >> 'NfsMirror'.. >> tunnel: -> sending command "disk" to remote >> tunnel: <- got reply >> 2023-10-10 18:08:49 volume 'CephCluster3Copie:vm-4980-disk-0' is >> 'NfsMirror:4980/vm-4980-disk-0.raw' on the target >> tunnel: -> sending command "config" to remote >> tunnel: <- got reply >> tunnel: -> sending command "start" to remote >> tunnel: <- got reply >> 2023-10-10 18:08:50 ERROR: online migrate failure - error - tunnel command >> '{"start_params":{"forcemachine":"pc-i440fx-8.0+pve0","forcecpu":null,"statefile":"unix","skiplock":1},"cmd":"start","migrate_opts":{"network":null,"nbd >> ":{"scsi0":{"volid":"NfsMirror:4980/vm-4980-disk-0.raw","success":true,"drivestr":"NfsMirror:4980/vm-4980-disk-0.raw,discard=on,format=raw,size=64G"}},"nbd_proto_version":1,"storagemap":{"default":"NfsMirror"},"migratedfrom":"n >> ode06-cluster4","type":"websocket","remote_node":"nodo01-cluster1","spice_ticket":null}}' >> failed - failed to handle 'start' command - start failed: QEMU exited with code >> 1 >> 2023-10-10 18:08:50 aborting phase 2 - cleanup resources >> 2023-10-10 18:08:50 migrate_cancel >> tunnel: -> sending command "stop" to remote >> tunnel: <- got reply >> tunnel: -> sending command "quit" to remote >> tunnel: <- got reply >> 2023-10-10 18:08:51 ERROR: migration finished with problems (duration 00:00:03) >> >> TASK ERROR: migration problems >> >> >> >> >> >> DESTINATION: >> >> mtunnel started >> received command 'version' >> received command 'bwlimit' >> received command 'disk' >> Formatting '/mnt/pve/NfsMirror/images/4980/vm-4980-disk-0.raw', fmt=raw >> size=68719476736 preallocation=off >> received command 'config' >> update VM 4980: -agent 1 -boot order=scsi0;ide2;net0 -cores 2 -ide2 >> none,media=cdrom -memory 8192 -name SeafileProTestS3 -net0 >> e1000=86:64:73:AB:33:AE,bridge=vlan902,tag=902 -numa 1 -ostype l26 -scsi0 >> NfsMirror:4980 >> /vm-4980-disk-0.raw,discard=on,format=raw,size=64G -scsihw virtio-scsi-pci >> -smbios1 uuid=39a07e5b-16b5-45a3-aad9-4e3f2b4e87ce -sockets 2 >> received command 'start' >> QEMU: vm vlans are not allowed on vnet vlan902 at >> /usr/share/perl5/PVE/Network/SDN/Zones/Plugin.pm line 228. >> QEMU: kvm: -netdev >> type=tap,id=net0,ifname=tap4980i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown: >> network script /var/lib/qemu-server/pve-bridge failed with status 6400 >> received command 'stop' >> received command 'quit' >> freeing volume 'NfsMirror:4980/vm-4980-disk-0.raw' as part of cleanup >> disk image '/mnt/pve/NfsMirror/images/4980/vm-4980-disk-0.raw' does not exist >> switching to exit-mode, waiting for client to disconnect >> mtunnel exited >> TASK OK >> >> >> Source VM conf file: >> >> agent: 1 >> balloon: 2048 >> boot: order=scsi0;ide2;net0 >> cores: 2 >> ide2: none,media=cdrom >> memory: 4096 >> name: SeafileProTestS3 >> net0: virtio=86:64:73:AB:33:AE,bridge=vmbr1,tag=902 >> numa: 1 >> ostype: l26 >> scsi0: CephCluster3Copie:vm-4980-disk-0,discard=on,size=64G >> scsihw: virtio-scsi-pci >> smbios1: uuid=39a07e5b-16b5-45a3-aad9-4e3f2b4e87ce >> sockets: 2 >> vmgenid: 035cd26d-c74e-405e-9b4d-481f26d9cf5f >> >> >> >> >>> Usually I would ask you to send me the entire journal but this is not >>> feasible on the mailing list. If necessary, I would recommend you open a >>> Thread in our community forum and I will take a look there. >> >>> https://forum.proxmox.com/ >>> >>> Best regards, >>> Stefan Lendl >> >> >> Thank you in advance, Fabrizio >> >> >>> >>>> Hello. >>>> I am testing qm remote-migrate with 2 pve 8.0.4 clusters. >>>> Source cluster has one bridge with vlan id on every VM, destination cluster uses >>>> SDN and a different bridge (vnet) without vlanid. >>>> If I migrate the vm, i need to specify both bridge and vlan-id, but I have not >>>> found an option to do it. >>>> >>>> PS: after migration, on the new cluster the vm is running without any problem, >>>> but on source cluster remains locked and in migration, so I need to issue a "qm >>>> unlock vmid" and stop/delete it. >>>> >>>> I know that is an experimental feature, so I send my test results. >>>> >>>> Regards, Fabrizio >>>> >>>> >>>> -- >>>> --- >>>> Fabrizio Cuseo - mailto:f.cuseo@panservice.it >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user@lists.proxmox.com >>> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> -- >> --- >> Fabrizio Cuseo - mailto:f.cuseo@panservice.it >> Direzione Generale - Panservice InterNetWorking >> Servizi Professionali per Internet ed il Networking >> Panservice e' associata AIIP - RIPE Local Registry >> Phone: +39 0773 410020 - Fax: +39 0773 470219 >> http://www.panservice.it mailto:info@panservice.it > > Numero verde nazionale: 800 901492 -- --- Fabrizio Cuseo - mailto:f.cuseo@panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:info@panservice.it Numero verde nazionale: 800 901492