From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C475C99B10 for ; Wed, 11 Oct 2023 11:38:28 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id A69D26A29 for ; Wed, 11 Oct 2023 11:37:58 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 11 Oct 2023 11:37:57 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 2821C449CE; Wed, 11 Oct 2023 11:37:57 +0200 (CEST) From: Stefan Lendl To: Fabrizio Cuseo Cc: Proxmox VE user list , pve-user In-Reply-To: <652060962.148106.1697010912527.JavaMail.zimbra@zimbra.panservice.it> References: <1200893807.110513.1696964512431.JavaMail.zimbra@zimbra.panservice.it> <87jzrtip2a.fsf@gmail.com> <652060962.148106.1697010912527.JavaMail.zimbra@zimbra.panservice.it> Date: Wed, 11 Oct 2023 11:37:55 +0200 Message-ID: <87zg0plcss.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain X-SPAM-LEVEL: Spam detection results: 0 AWL 0.060 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] qm remote-migrate X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Oct 2023 09:38:28 -0000 Fabrizio Cuseo writes: Thanks for providing the details. I will investigate the situation and we will consider a solution for our upcoming SDN upgrade. As a solution for now, please try to remove the VLAN tag from the source VM and try to migrate again. The target net interface will not require a VLAN tag assigned (and is therefore not allowed) on the VM because it is configured via SDN already. Best regards, Stefan > ----- Il 11-ott-23, alle 9:41, Stefan Lendl s.lendl@proxmox.com ha scritto: > >> Fabrizio Cuseo writes: >> >> Hello Fabrizio, >> >> To better understand your issue, the source cluster has a VM with a >> bridge with a VLAN tag assigned and the target cluster does not have the >> same setup but uses SDN (vnet) without vlan. > > Yes, it's correct. > > >> After migration you manually changed the VMs configuration to match the >> new setup? > > I can't because remote-migration returns an error (I cannot specify vlan tag on that bridge) > >> What SDN configuration are you using on the traget cluster? >> Please send the output of the following: >> >> head -n -1 /etc/pve/sdn/*.cfg > > Can I send you in private ? Because is full of customer's names :/ > > But, this is a part of files: > > ==> /etc/pve/sdn/zones.cfg <== > vlan: ZonaVLAN > bridge vmbr0 > ipam pve > > qinq: VPVT > bridge vmbr0 > tag 929 > ipam pve > vlan-protocol 802.1q > > > ==> /etc/pve/sdn/vnets.cfg <== > vnet: test100 > zone FWHous > alias Vlan 100 Test 921 qinq > tag 100 > > vnet: vlan902 > zone ZonaVLAN > alias Vlan 902 Private-Vlan > tag 902 > > > > >> What was to exact command you ran to start the remote-migrate process? > > qm remote-migrate 4980 4980 'host=172.16.20.41,apitoken=PVEAPIToken=root@pam!remotemigrate=hiddensecret,fingerprint=hiddenfingerprint' --target-bridge vlan902 --target-storage NfsMirror --online > > > >> Did you notice any suspicios log messages in the source clusters >> journal? > > Source: > > tunnel: -> sending command "version" to remote > tunnel: <- got reply > 2023-10-10 18:08:48 local WS tunnel version: 2 > 2023-10-10 18:08:48 remote WS tunnel version: 2 > 2023-10-10 18:08:48 minimum required WS tunnel version: 2 > websocket tunnel started > 2023-10-10 18:08:48 starting migration of VM 4980 to node 'nodo01-cluster1' (172.16.20.41) > tunnel: -> sending command "bwlimit" to remote > tunnel: <- got reply > 2023-10-10 18:08:49 found local disk 'CephCluster3Copie:vm-4980-disk-0' (attached) > 2023-10-10 18:08:49 mapped: net0 from vmbr1 to vlan902 > 2023-10-10 18:08:49 Allocating volume for drive 'scsi0' on remote storage 'NfsMirror'.. > tunnel: -> sending command "disk" to remote > tunnel: <- got reply > 2023-10-10 18:08:49 volume 'CephCluster3Copie:vm-4980-disk-0' is 'NfsMirror:4980/vm-4980-disk-0.raw' on the target > tunnel: -> sending command "config" to remote > tunnel: <- got reply > tunnel: -> sending command "start" to remote > tunnel: <- got reply > 2023-10-10 18:08:50 ERROR: online migrate failure - error - tunnel command '{"start_params":{"forcemachine":"pc-i440fx-8.0+pve0","forcecpu":null,"statefile":"unix","skiplock":1},"cmd":"start","migrate_opts":{"network":null,"nbd > ":{"scsi0":{"volid":"NfsMirror:4980/vm-4980-disk-0.raw","success":true,"drivestr":"NfsMirror:4980/vm-4980-disk-0.raw,discard=on,format=raw,size=64G"}},"nbd_proto_version":1,"storagemap":{"default":"NfsMirror"},"migratedfrom":"n > ode06-cluster4","type":"websocket","remote_node":"nodo01-cluster1","spice_ticket":null}}' failed - failed to handle 'start' command - start failed: QEMU exited with code 1 > 2023-10-10 18:08:50 aborting phase 2 - cleanup resources > 2023-10-10 18:08:50 migrate_cancel > tunnel: -> sending command "stop" to remote > tunnel: <- got reply > tunnel: -> sending command "quit" to remote > tunnel: <- got reply > 2023-10-10 18:08:51 ERROR: migration finished with problems (duration 00:00:03) > > TASK ERROR: migration problems > > > > > > DESTINATION: > > mtunnel started > received command 'version' > received command 'bwlimit' > received command 'disk' > Formatting '/mnt/pve/NfsMirror/images/4980/vm-4980-disk-0.raw', fmt=raw size=68719476736 preallocation=off > received command 'config' > update VM 4980: -agent 1 -boot order=scsi0;ide2;net0 -cores 2 -ide2 none,media=cdrom -memory 8192 -name SeafileProTestS3 -net0 e1000=86:64:73:AB:33:AE,bridge=vlan902,tag=902 -numa 1 -ostype l26 -scsi0 NfsMirror:4980 > /vm-4980-disk-0.raw,discard=on,format=raw,size=64G -scsihw virtio-scsi-pci -smbios1 uuid=39a07e5b-16b5-45a3-aad9-4e3f2b4e87ce -sockets 2 > received command 'start' > QEMU: vm vlans are not allowed on vnet vlan902 at /usr/share/perl5/PVE/Network/SDN/Zones/Plugin.pm line 228. > QEMU: kvm: -netdev type=tap,id=net0,ifname=tap4980i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown: network script /var/lib/qemu-server/pve-bridge failed with status 6400 > received command 'stop' > received command 'quit' > freeing volume 'NfsMirror:4980/vm-4980-disk-0.raw' as part of cleanup > disk image '/mnt/pve/NfsMirror/images/4980/vm-4980-disk-0.raw' does not exist > switching to exit-mode, waiting for client to disconnect > mtunnel exited > TASK OK > > > Source VM conf file: > > agent: 1 > balloon: 2048 > boot: order=scsi0;ide2;net0 > cores: 2 > ide2: none,media=cdrom > memory: 4096 > name: SeafileProTestS3 > net0: virtio=86:64:73:AB:33:AE,bridge=vmbr1,tag=902 > numa: 1 > ostype: l26 > scsi0: CephCluster3Copie:vm-4980-disk-0,discard=on,size=64G > scsihw: virtio-scsi-pci > smbios1: uuid=39a07e5b-16b5-45a3-aad9-4e3f2b4e87ce > sockets: 2 > vmgenid: 035cd26d-c74e-405e-9b4d-481f26d9cf5f > > > > >> Usually I would ask you to send me the entire journal but this is not >> feasible on the mailing list. If necessary, I would recommend you open a >> Thread in our community forum and I will take a look there. > >> https://forum.proxmox.com/ >> >> Best regards, >> Stefan Lendl > > > Thank you in advance, Fabrizio > > >> >>> Hello. >>> I am testing qm remote-migrate with 2 pve 8.0.4 clusters. >>> Source cluster has one bridge with vlan id on every VM, destination cluster uses >>> SDN and a different bridge (vnet) without vlanid. >>> If I migrate the vm, i need to specify both bridge and vlan-id, but I have not >>> found an option to do it. >>> >>> PS: after migration, on the new cluster the vm is running without any problem, >>> but on source cluster remains locked and in migration, so I need to issue a "qm >>> unlock vmid" and stop/delete it. >>> >>> I know that is an experimental feature, so I send my test results. >>> >>> Regards, Fabrizio >>> >>> >>> -- >>> --- >>> Fabrizio Cuseo - mailto:f.cuseo@panservice.it >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user@lists.proxmox.com >> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > -- > --- > Fabrizio Cuseo - mailto:f.cuseo@panservice.it > Direzione Generale - Panservice InterNetWorking > Servizi Professionali per Internet ed il Networking > Panservice e' associata AIIP - RIPE Local Registry > Phone: +39 0773 410020 - Fax: +39 0773 470219 > http://www.panservice.it mailto:info@panservice.it > Numero verde nazionale: 800 901492