From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <aderumier@odiso.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 9A32B6A34B for <pve-devel@pve.proxmox.com>; Fri, 22 Jan 2021 15:34:54 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9164A1DA4F for <pve-devel@pve.proxmox.com>; Fri, 22 Jan 2021 15:34:54 +0100 (CET) Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 5D54A1DA42 for <pve-devel@pve.proxmox.com>; Fri, 22 Jan 2021 15:34:52 +0100 (CET) Received: by mail-wm1-x32f.google.com with SMTP id s24so6807170wmj.0 for <pve-devel@pve.proxmox.com>; Fri, 22 Jan 2021 06:34:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=odiso-com.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:date:user-agent:mime-version :content-transfer-encoding; bh=CTa7uaqK9S5otV0MNWKydJFIeyQ2oqEO6zpBqujJaek=; b=s3sLIy8U0TPCqNUriu9X9GLH4M5uZRBcOhALO7+Kp1Sp+r8NguSH1gmeQv1NCK2XKK GtLBAXY2r6VHN8SebOKRk8rZqhWXxApNen0Hv6KYg3NJXDHJlH4b0Xmje5StiX08TQbN bHb00ppxwQmhym69Xo/bqFob0qbFvHg5BUJOIlmarbwKtLz600FfGreIHLnIUWq7QqOI L0IgGlD+Y4KzNIpEAnpknGdJJ/P1+2gF7KnFY0QhmkwA18UkVZoe7khnJTm8UTALQi1J FTr0uL9aQRkIIgOyQMUU+jaIrUtNeJZgks6KMsYY8JPKe46SZObY5wvmKSEsgDxr5N+0 xXXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:user-agent :mime-version:content-transfer-encoding; bh=CTa7uaqK9S5otV0MNWKydJFIeyQ2oqEO6zpBqujJaek=; b=G/9LKYtHfe/cAhy8ZkECwjC3qeMCc0NMPTCJYPX+XRYxWTlqDsoS+pbWnk0taiahHO bb84qJc8iuA7vfUo6ZqFjOjXPXAZoU46W1A4FsWBex9o+mxAWWzaERRex+alMpR0H0Zr a3TdWj+LMHV/Xtn7tVYRlF5+aBs37fnSXFTMn4WYtJ0EATspHluhCWxo28ZHS0HRtfB4 ejg6FGxso1x4/HB++eyOjvxOEXXEYxy2u9pc4eyXTII+PX30p4p1HLgSsZrodO/AdSAV 9NzL9HLSP6AN+x/nzJvoUpo57QxVXYbk+V9Xh0T3ZEkb1oDLqIgi/kBoAjtmdJjeY6+6 KZjA== X-Gm-Message-State: AOAM533vYUrW9IG4kPLsyGdRZM+Bh20LNGEU0s4MlWALjYxD1shblhpV DEtPFO8o7CrEpRcujroPRwuyZUVGsdq+eUel X-Google-Smtp-Source: ABdhPJzMjBWRcFcSObdDbXDFKiCotGEDrufS4hQ0xNgbg2APW5vlb0a7w3X5c9Ixxe2bpKvOjUcfhQ== X-Received: by 2002:a1c:4907:: with SMTP id w7mr4275841wma.118.1611326091701; Fri, 22 Jan 2021 06:34:51 -0800 (PST) Received: from ?IPv6:2a0a:1580:0:1::100c? (ovpn1.odiso.net. [2a0a:1580:2000::3f]) by smtp.gmail.com with ESMTPSA id p18sm11497847wmc.31.2021.01.22.06.34.51 for <pve-devel@pve.proxmox.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 06:34:51 -0800 (PST) Message-ID: <b08a69e25cdfe7615bd192ab07b169a0100fadeb.camel@odiso.com> From: aderumier@odiso.com To: pve-devel <pve-devel@pve.proxmox.com> Date: Fri, 22 Jan 2021 15:34:50 +0100 Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.3 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.217 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] qemu live migration: bigger downtime recently X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Fri, 22 Jan 2021 14:34:54 -0000 Hi, I have notice recently bigger downtime on qemu live migration. (I'm not sure if it's after qemu update or qemu-server update) migration: type=insecure qemu-server 6.3-2 pve-qemu-kvm 5.1.0-7 (I'm not sure about the machine running qemu version) Here a sample: 2021-01-22 15:28:38 starting migration of VM 226 to node 'kvm13' (10.3.94.70) 2021-01-22 15:28:42 starting VM 226 on remote node 'kvm13' 2021-01-22 15:28:44 start remote tunnel 2021-01-22 15:28:45 ssh tunnel ver 1 2021-01-22 15:28:45 starting online/live migration on tcp:10.3.94.70:60000 2021-01-22 15:28:45 set migration_caps 2021-01-22 15:28:45 migration speed limit: 8589934592 B/s 2021-01-22 15:28:45 migration downtime limit: 100 ms 2021-01-22 15:28:45 migration cachesize: 268435456 B 2021-01-22 15:28:45 set migration parameters 2021-01-22 15:28:45 start migrate command to tcp:10.3.94.70:60000 2021-01-22 15:28:47 migration speed: 1024.00 MB/s - downtime 2117 ms 2021-01-22 15:28:47 migration status: completed 2021-01-22 15:28:51 migration finished successfully (duration 00:00:13) TASK OK That's strange because I don't see the memory transfert loop logs Migrate back to original host is working 2021-01-22 15:29:34 starting migration of VM 226 to node 'kvm2' (::ffff:10.3.94.50) 2021-01-22 15:29:36 starting VM 226 on remote node 'kvm2' 2021-01-22 15:29:39 start remote tunnel 2021-01-22 15:29:40 ssh tunnel ver 1 2021-01-22 15:29:40 starting online/live migration on tcp:[::ffff:10.3.94.50]:60000 2021-01-22 15:29:40 set migration_caps 2021-01-22 15:29:40 migration speed limit: 8589934592 B/s 2021-01-22 15:29:40 migration downtime limit: 100 ms 2021-01-22 15:29:40 migration cachesize: 268435456 B 2021-01-22 15:29:40 set migration parameters 2021-01-22 15:29:40 start migrate command to tcp:[::ffff:10.3.94.50]:60000 2021-01-22 15:29:41 migration status: active (transferred 396107554, remaining 1732018176), total 2165383168) 2021-01-22 15:29:41 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0 2021-01-22 15:29:42 migration status: active (transferred 973010921, remaining 1089216512), total 2165383168) 2021-01-22 15:29:42 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0 2021-01-22 15:29:43 migration status: active (transferred 1511925476, remaining 483463168), total 2165383168) 2021-01-22 15:29:43 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0 2021-01-22 15:29:44 migration speed: 512.00 MB/s - downtime 148 ms 2021-01-22 15:29:44 migration status: completed 2021-01-22 15:29:47 migration finished successfully (duration 00:00:13) TASK OK Then migrate it again like the first migration is working too 2021-01-22 15:31:07 starting migration of VM 226 to node 'kvm13' (10.3.94.70) 2021-01-22 15:31:10 starting VM 226 on remote node 'kvm13' 2021-01-22 15:31:12 start remote tunnel 2021-01-22 15:31:13 ssh tunnel ver 1 2021-01-22 15:31:13 starting online/live migration on tcp:10.3.94.70:60000 2021-01-22 15:31:13 set migration_caps 2021-01-22 15:31:13 migration speed limit: 8589934592 B/s 2021-01-22 15:31:13 migration downtime limit: 100 ms 2021-01-22 15:31:13 migration cachesize: 268435456 B 2021-01-22 15:31:13 set migration parameters 2021-01-22 15:31:13 start migrate command to tcp:10.3.94.70:60000 2021-01-22 15:31:14 migration status: active (transferred 1092088188, remaining 944365568), total 2165383168) 2021-01-22 15:31:14 migration xbzrle cachesize: 268435456 transferred 0 pages 0 cachemiss 0 overflow 0 2021-01-22 15:31:15 migration speed: 1024.00 MB/s - downtime 55 ms 2021-01-22 15:31:15 migration status: completed 2021-01-22 15:31:19 migration finished successfully (duration 00:00:12) TASK OK Any idea ? Maybe a specific qemu version bug ?