From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6A660737DE for ; Fri, 16 Apr 2021 09:36:54 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 57FD721698 for ; Fri, 16 Apr 2021 09:36:24 +0200 (CEST) Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 533342168E for ; Fri, 16 Apr 2021 09:36:23 +0200 (CEST) Received: by mail-wm1-x334.google.com with SMTP id f195-20020a1c1fcc0000b029012eb88126d7so3697302wmf.3 for ; Fri, 16 Apr 2021 00:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=odiso-com.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=gYfoDadJUBPvx0FW6lcxJkkuZj1rP+bpNBxKpkkWhnE=; b=r31u8mvp4/Qnhh80mymJK/XzwkbdjnjQX9lpKHwHiayIj5zc6F6Bwft/x5FR8dG1GC jxldt0j9PfMoV+UFCag6Y8cIGe5enujG0i7ngIKCLqlrxokqFJmnQ4Lh0QikMG5XFkHW sNOTya6m+khjzc7+NMMIiBOoZURev48i39gxHtqhqnH2y8yv4Pb90g5JkmSWNGFE1Vlp wHDGkpkbumwlmhVumjgAWMz6RLWWNQkc+RVetXALxLPtO21kMCdyclHPFz/xObbbx3fR RFbqGddxs25jWGsbCEQ5L26k34uPdyR7ZFnRMxwQA1UshK/OXeGqNMxQgO/WXDQ6R1rp taOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=gYfoDadJUBPvx0FW6lcxJkkuZj1rP+bpNBxKpkkWhnE=; b=Z3ZGzKusGwiD2h5R4Ie19cYiDkvg5XEJZf1gsse6DdKJuBztEOZxHcDTscaBpA9tFQ VDFJFW2zilqnip7jHcCEpKm7SBiYrG23zeH09Uo99zDaTh+1WJy0sYCv9Huv9lEjLBGj HznpgkzBH9JK09EcGPkIwpWm0O/36Lqo7+y30PY/H+XjJzFi6EfqlDCAEK5tgaCRG42K yTNNQQOWpYtqrRviTcUHxyeT8cI0BOqvOwkpIl6+PQ2qqk1lqBCI5eNdQ7Qvr+n0zTDr ewlR42MNOcYE1mvO2imkNVAviwg/3tZTA3E0vI3Iv0QnIS4pE+PzRirGx6nx87wZ7hhI nicg== X-Gm-Message-State: AOAM531BPYfY5anIfaylJ2Ixeh4c1URf4Zn834UDbEMl7H0F9kSl+eVO /AqC5+2C4EZlskhQ/HQkeyDba7vRyYgVYQ== X-Google-Smtp-Source: ABdhPJxNfYLyZhGJ5l/OdhONX/GZC59hweG1hiYOXi5N3yAkrOObHKwkvbRyU1TznMi4CjtOoYqxJA== X-Received: by 2002:a05:600c:4b92:: with SMTP id e18mr6984457wmp.150.1618558576800; Fri, 16 Apr 2021 00:36:16 -0700 (PDT) Received: from [192.168.178.50] ([79.132.252.54]) by smtp.gmail.com with ESMTPSA id g3sm8829629wrp.46.2021.04.16.00.36.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 16 Apr 2021 00:36:16 -0700 (PDT) To: =?UTF-8?Q?Fabian_Gr=c3=bcnbichler?= , Proxmox VE development discussion References: <20210413121640.3602975-1-f.gruenbichler@proxmox.com> <1618496842.5t56y2jruz.astroid@nora.none> From: alexandre derumier Message-ID: <20079a8d-070a-dec3-cbfb-80957b068b2e@odiso.com> Date: Fri, 16 Apr 2021 09:36:15 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <1618496842.5t56y2jruz.astroid@nora.none> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-SPAM-LEVEL: Spam detection results: 0 AWL 0.125 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] [RFC qemu-server++ 0/22] remote migration X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Apr 2021 07:36:54 -0000 >>looking forward to feedback :) you'll need to put the >>proxmox-websocket-tunnel binary into $PATH of pveproxy/qm, after >>building it with 'cargo build'. oh, I didn't known that the rust version was already ready. great :) I'll try to do my first rust build :) (I really need to learn it, seem to be a great language) >>if your inter-DC link is fast enough, you'll likely be hit by the >>pveproxy bottleneck. it would still be interesting to get some >>real-world numbers, I haven't tested with baremetal and fast storage >>yet. yes, I have enough bandwidth (25gbs), so no problem here. I'll try with && without any storage to compare. (as I'm using ceph with nvme, I think it'll be the bottleneck with qemu mirror) >>please be aware that this is very much experimental code still! yes, sure , no problem ;) On 15/04/2021 16:32, Fabian Grünbichler wrote: > On April 15, 2021 4:04 pm, alexandre derumier wrote: >> Hi, >> >> thanks for working on this ! >> >> I'll be able to test it soon as I'll need to migrate 200-300 vms between >> 2 datacenter soon. > looking forward to feedback :) you'll need to put the > proxmox-websocket-tunnel binary into $PATH of pveproxy/qm, after > building it with 'cargo build'. > > if your inter-DC link is fast enough, you'll likely be hit by the > pveproxy bottleneck. it would still be interesting to get some > real-world numbers, I haven't tested with baremetal and fast storage > yet. > > please be aware that this is very much experimental code still! > >> I think it could be great to add optionnal "tag" option to targetbridge, >> as it could be different on target cluster. > hmm, we could have another (optional) map for VLAN tags? since tags and > bridges are not one entity (you can have on interface on bridge A with > tag X, and another interface on bridge A with tag Y, and those need to > be mapped to bridge B with tag P and bridge B with tag Q, for example). > >> Also, we should transfert vm firewall config. > yes, that's definitely true. another source of potential > mismatches/things to check before migrating (security groups/aliases!) > >> On 13/04/2021 14:16, Fabian Grünbichler wrote: >>> this series adds remote migration for VMs. there's still plenty of >>> TODOs/FIXMEs/stuff that requires discussion, hence the RFC. live >>> migration with NBD and storage-migrated disks should work already. >>> >>> the performance bottle neck (~190MB/s on loopback) for the websocket >>> connection seems to be in pveproxy at the moment - the rust code should >>> manage about 700MB/s. >>> >>> overview over affected repos and changes, see individual patches for >>> more details. >>> >>> proxmox: >>> >>> some compatible changes to make websocket code usable for client-side >>> connections, required by proxmox-websocket-tunnel >>> >>> proxmox-websocket-tunnel: >>> >>> new tunnel helper tool for forwarding commands and data over websocket >>> connections, required by qemu-server on source side >>> TODO: better error handling >>> TODO: fingerprint checking/valid certs/.. >>> TODO: WS key generation >>> TODO: decide on mask? >>> TODO: investigate performance bottlenecks once PVE api server gets >>> faster >>> >>> pve-access-control: >>> >>> new ticket type, required by qemu-server on target side >>> >>> pve-cluster: >>> >>> new remote.cfg and related helpers, required by qemu-server on source >>> side >>> TODO: ACLs, CLI, API for managing config >>> TODO: handling of discovered nodes with valid certificates >>> TODO: add additional information like default bwlimits, storage/bridge >>> mappings >>> >>> pve-common: >>> >>> bridgepair format akin to storage pair, pve-bridge-id option, required >>> by qemu-server >>> TODO: adapt pve-container >>> >>> pve-guest-common: >>> >>> handle remote migration (no SSH) in AbstractMigrate, >>> required by qemu-server >>> >>> pve-manager: >>> >>> new 'addr' endpoint for retrieving remote node IPs, required on target >>> node >>> >>> pve-storage: >>> >>> extend 'pvesm import' to allow import from UNIX socket, required on >>> target node by qemu-server >>> >>> qemu-server: >>> >>> some refactoring, new mtunnel endpoints, new remote_migration endpoints >>> TODO: check remote ACLs >>> TODO: handle pending changes and snapshots >>> TODO: CLI for remote migration >>> potential TODO: expose remote info via additional endpoints (resources? vmids? >>> permissions? ...) >>> >>> as usual, some of the patches are best viewed with '-w', especially in >>> qemu-server.. >>> >>> >>> _______________________________________________ >>> pve-devel mailing list >>> pve-devel@lists.proxmox.com >>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >>> >>