From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id 7E99A1FF191
for <inbox@lore.proxmox.com>; Mon, 2 Jun 2025 10:35:54 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 725B51D277;
Mon, 2 Jun 2025 10:36:10 +0200 (CEST)
References: <mailman.538.1747833190.394.pve-devel@lists.proxmox.com>
<1283184248.17536.1747895442851@webmail.proxmox.com>
<857cbd6c-6866-417d-a71f-f5b5297bf09c@storpool.com>
<1349127939.17705.1747902137180@webmail.proxmox.com>
<CAHXTzuk7tYRJV_j=88RWc3R3C7AkiEdFUXi88m5qwnDeYDEC+A@mail.gmail.com>
<11746909.21389.1748414016786@webmail.proxmox.com>
<CAHXTzumXeyJQQCj+45Hmy5qdU+BTFBYbHVgPy0u3VS-qS=_bDQ@mail.gmail.com>
<1695649345.530.1748849837156@webmail.proxmox.com>
In-Reply-To: <1695649345.530.1748849837156@webmail.proxmox.com>
Date: Mon, 2 Jun 2025 11:35:22 +0300
To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= <f.gruenbichler@proxmox.com>
MIME-Version: 1.0
Message-ID: <mailman.145.1748853369.395.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Denis Kanchev via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Denis Kanchev <denis.kanchev@storpool.com>,
Wolfgang Bumiller <w.bumiller@proxmox.com>,
Proxmox VE development discussion <pve-devel@lists.proxmox.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: Re: [pve-devel] PVE child process behavior question
Content-Type: multipart/mixed; boundary="===============0806721311762656938=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
--===============0806721311762656938==
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <denis.kanchev@storpool.com>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 8F91DCAD16
for <pve-devel@lists.proxmox.com>; Mon, 2 Jun 2025 10:36:08 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 6AE311D2C4
for <pve-devel@lists.proxmox.com>; Mon, 2 Jun 2025 10:36:08 +0200 (CEST)
Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com [IPv6:2607:f8b0:4864:20::112c])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Mon, 2 Jun 2025 10:36:06 +0200 (CEST)
Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-70e5e6ab7b8so36274947b3.1
for <pve-devel@lists.proxmox.com>; Mon, 02 Jun 2025 01:36:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=storpool.com; s=google; t=1748853359; x=1749458159; darn=lists.proxmox.com;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:from:to:cc:subject:date:message-id:reply-to;
bh=mG/BoCEcj4jzrG6N+TSD7EO1zWEkdM8d0SCBe9CyRZU=;
b=LwY4eMHSrArZoqq9EmGCDT2BkSLYcS7mu2iGvaddXD39pTBhFt6sFmWgudtIDNEY3F
xlYfppaXWY1DXHHz26Kllo0Pyo8XVrIFGUwRCBHnPHXboojz115ZqKralOzdfWYj2EQm
oMHCQq+XJxJfePACVY4LDkVg5TaARn1cYt2iybuHPaOuk+8Olkq7QhbUEzeznATau2ws
Tp40A874pLikhDWCe3ph8lBLtiywk/Hjff+YItN8gL8HimaB5lf5opSYrRI9cM0PsslX
KUu9VeuYc5W3K9H7dusuRciEyNjCi25qU7wB67hkUO0sPpKv+jMCBwt5Ttures0y52R4
r/DA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1748853359; x=1749458159;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=mG/BoCEcj4jzrG6N+TSD7EO1zWEkdM8d0SCBe9CyRZU=;
b=lzKgi2NWP0o3yZFh4dp1rpPftiu5HpA1obO5Gp6eueA/vTP20rgKPras743EsF4wBK
/VYoOIJlA5ppV+Z5ZPJpNIQE9cHCVL2cKlpJ4RV10cvx8wJ8xDfXQYhcmvljirMv8Wq7
NhFBPt83lLJ9c/lLb7hv+XmvuSKWhDqwwCdzMrJO6/DdVvU/rVa0H/vZy0HecQ5YqFjh
+vp6YDznPPzbNeADv4SKwZSIKmIbzZ3dsAmbzPnbkg6wzmtkbhSIUoqy2td2AwhO+B73
JFz/Q+zNrdsQgtRG5AigMDfs7Ey0jWVAPe8AlCuibtJgGLTZ2o/4jnBfcsA3L6uA4pD0
oDtg==
X-Gm-Message-State: AOJu0YyXZeoptKzRhQr4eVFpg+kTWIfkyK9fBDBDG6BS9Rv0bFFK63NA
Efla35Siwt7S9dK+NvVhCvLJCB2Q4onDxpccl9iDwJoQz8N6nzQL7XzHI8EL8dZtz07FGBODeHb
v3Hg2zwy1Ao+e0hClota5yADLZFnijJIXilo1wKrT4w==
X-Gm-Gg: ASbGnctNY8BkvNyTuWz4KGMAPSNeY6spMg1enHjwuLwGimdd7NngQ78fmihaDUk1j6w
lhl2aQnzlWMf//2SaKPvl8TGaokzJLg8D6tZZwMQV39y4bFwC5dH0heLPE1nUogZ/vTtc0kzRjh
h8Pr8Xdgn4C534QVNMNyc1vXYOeWUMSl5CuA==
X-Google-Smtp-Source: AGHT+IEhFQTZlwkbB47NrkQn1pI4zJXDUO8g3AM3si84hrxZaAG86R6ujismU584bc8kGsYFzipO3EyclBycWff2NX4=
X-Received: by 2002:a05:690c:b9b:b0:70e:53a1:a814 with SMTP id
00721157ae682-70f97f2a5f3mr163884717b3.29.1748853359090; Mon, 02 Jun 2025
01:35:59 -0700 (PDT)
MIME-Version: 1.0
References: <mailman.538.1747833190.394.pve-devel@lists.proxmox.com>
<1283184248.17536.1747895442851@webmail.proxmox.com> <857cbd6c-6866-417d-a71f-f5b5297bf09c@storpool.com>
<1349127939.17705.1747902137180@webmail.proxmox.com> <CAHXTzuk7tYRJV_j=88RWc3R3C7AkiEdFUXi88m5qwnDeYDEC+A@mail.gmail.com>
<11746909.21389.1748414016786@webmail.proxmox.com> <CAHXTzumXeyJQQCj+45Hmy5qdU+BTFBYbHVgPy0u3VS-qS=_bDQ@mail.gmail.com>
<1695649345.530.1748849837156@webmail.proxmox.com>
In-Reply-To: <1695649345.530.1748849837156@webmail.proxmox.com>
From: Denis Kanchev <denis.kanchev@storpool.com>
Date: Mon, 2 Jun 2025 11:35:22 +0300
X-Gm-Features: AX0GCFt7KpZjvXlECBsUtUV_6tzODlB8XMgUW6QDgS-3aEO2wI-Z6ZyB3waganQ
Message-ID: <CAHXTzukAMG9050Ynn-KRSqhCz2Y0m6vnAQ7FEkCmEdQT3HapfQ@mail.gmail.com>
Subject: Re: [pve-devel] PVE child process behavior question
To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= <f.gruenbichler@proxmox.com>
Cc: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Wolfgang Bumiller <w.bumiller@proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid
DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain
DMARC_PASS -0.1 DMARC pass policy
HTML_MESSAGE 0.001 HTML included in message
RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [storpool.com]
Content-Type: text/plain; charset="UTF-8"
X-Content-Filtered-By: Mailman/MimeDel 2.1.29
> I thought your storage plugin is a shared storage, so there is no storage
migration at all, yet you keep talking about storage migration?
It's a shared storage indeed, the issue was that the migration process on
the destination host got OOM killed and the migration failed, most probably
that's why there is no log about the storage migration, but that didn't
stop the storage migration on the destination host.
2025-04-11T03:26:52.283913+07:00 telpr01pve03 kernel: [96031.290519] pvesh
invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Here is one more migration task attempt where it lived long enough to show
more detailed log:
2025-04-11 03:29:11 starting migration of VM 2421 to node 'telpr01pve06'
(10.10.17.6)
2025-04-11 03:29:11 starting VM 2421 on remote node 'telpr01pve06'
2025-04-11 03:29:15 [telpr01pve06] Warning: sch_htb: quantum of class 10001
is big. Consider r2q change.
2025-04-11 03:29:15 [telpr01pve06] kvm: failed to find file
'/usr/share/qemu-server/bootsplash.jpg'
2025-04-11 03:29:15 start remote tunnel
2025-04-11 03:29:16 ssh tunnel ver 1
2025-04-11 03:29:16 starting online/live migration on
unix:/run/qemu-server/2421.migrate
2025-04-11 03:29:16 set migration capabilities
2025-04-11 03:29:16 migration downtime limit: 100 ms
2025-04-11 03:29:16 migration cachesize: 256.0 MiB
2025-04-11 03:29:16 set migration parameters
2025-04-11 03:29:16 start migrate command to
unix:/run/qemu-server/2421.migrate
2025-04-11 03:29:17 migration active, transferred 281.0 MiB of 2.0 GiB
VM-state, 340.5 MiB/s
2025-04-11 03:29:18 migration active, transferred 561.5 MiB of 2.0 GiB
VM-state, 307.2 MiB/s
2025-04-11 03:29:19 migration active, transferred 849.2 MiB of 2.0 GiB
VM-state, 288.5 MiB/s
2025-04-11 03:29:20 migration active, transferred 1.1 GiB of 2.0 GiB
VM-state, 283.7 MiB/s
2025-04-11 03:29:21 migration active, transferred 1.4 GiB of 2.0 GiB
VM-state, 302.5 MiB/s
2025-04-11 03:29:23 migration active, transferred 1.8 GiB of 2.0 GiB
VM-state, 278.6 MiB/s
2025-04-11 03:29:23 migration status error: failed
2025-04-11 03:29:23 ERROR: online migrate failure - aborting
2025-04-11 03:29:23 aborting phase 2 - cleanup resources
2025-04-11 03:29:23 migrate_cancel
2025-04-11 03:29:25 ERROR: migration finished with problems (duration
00:00:14)
TASK ERROR: migration problems
> could you provide the full migration task log and the VM config?
2025-04-11 03:26:50 starting migration of VM 2421 to node 'telpr01pve03'
(10.10.17.3) ### QemuMigrate::phase1() +749
2025-04-11 03:26:50 starting VM 2421 on remote node 'telpr01pve03' #
QemuMigrate::phase2_start_local_cluster() +888
2025-04-11 03:26:52 ERROR: online migrate failure - remote command failed
with exit code 255
2025-04-11 03:26:52 aborting phase 2 - cleanup resources
2025-04-11 03:26:52 migrate_cancel
2025-04-11 03:26:53 ERROR: migration finished with problems (duration
00:00:03)
TASK ERROR: migration problems
VM config
#Ubuntu-24.04-14082024
#StorPool adjustment
agent: 1,fstrim_cloned_disks=1
autostart: 1
boot: c
bootdisk: scsi0
cipassword: XXX
citype: nocloud
ciupgrade: 0
ciuser: test
cores: 2
cpu: EPYC-Genoa
cpulimit: 2
ide0: VMDataSp:vm-2421-cloudinit.raw,media=cdrom
ipconfig0: ipxxx
memory: 2048
meta: creation-qemu=8.1.5,ctime=1722917972
name: kredibel-service
nameserver: xxx
net0: virtio=xxx,bridge=vmbr2,firewall=1,rate=250,tag=220
numa: 0
onboot: 1
ostype: l26
scsi0:
VMDataSp:vm-2421-disk-0-sp-bj7n.b.sdj.raw,aio=native,discard=on,iops_rd=20000,iops_rd_max=40000,iops_rd_max_length=60,iops_wr=20000,iops_wr_max=40000,iops_wr_max_length=60,iothread=1,size=40G
scsihw: virtio-scsi-single
searchdomain: neo.internal
serial0: socket
smbios1: uuid=dfxxx
sockets: 1
sshkeys: ssh-rsa%
vmgenid: 17b154a0-
IN this case the call to PVE::Storage::Plugin::activate_volume() was
performed after migration cancelation
2025-04-11T03:26:53.072206+07:00 telpr01pve03 qm[3670228]: StorPool plugin:
NOT a live migration of VM 2421, will force detach volume ~bj7n.b.abe <<<
This log is from the sub activate_volume() in our custom storage plugin
--===============0806721311762656938==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--===============0806721311762656938==--