From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 740F068DF6 for ; Tue, 9 Mar 2021 21:50:56 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5DCC6EB81 for ; Tue, 9 Mar 2021 21:50:26 +0100 (CET) Received: from mout.web.de (mout.web.de [212.227.15.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 7CC6CEB6A for ; Tue, 9 Mar 2021 21:50:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1615323018; bh=iYg8B9b8Vh4RgFONS0oKmQ+B1Lgq1w13W0eZS7uZZk8=; h=X-UI-Sender-Class:From:Subject:To:Date; b=a29Dnn7mc3SWZDnPm9mpLYhPVOnEhErVEl+coPYAiXP7QlYP3PQ5XlHHguXFPkdvJ Xns8jUQaYd3c539w3Q7XrrvgJE5/OFXQjrnXIIvtZP0uIASozHQXEoboq2nihfc46r DRd2+68X7wLcUiprBapezcUyMVJlbWKDw8zUQtq0= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from Rolands-MacBook-Pro.cybercon.de ([79.216.115.8]) by smtp.web.de (mrweb005 [213.165.67.108]) with ESMTPSA (Nemesis) id 1MCGSU-1lS0hS1JvG-009mWX for ; Tue, 09 Mar 2021 21:45:08 +0100 From: Roland To: pve-devel@lists.proxmox.com Message-ID: <792c380c-6757-e058-55f6-f7d5436417f9@web.de> Date: Tue, 9 Mar 2021 21:45:11 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 X-Provags-ID: V03:K1:8cbKheuQ8KQazy+6KVrHZZ/oH7cJwygi0LP1JMR8L9QmrcIIKQ0 J417n4WeXWcEYJ46sMsu48jXzdMMfcilEaQ1CdPOH8jknm4h1QuVdubL6bNz0F6DujX8ODC 1b7jyZZ0wB0SVOTnMqNx1WJB6417LnuJgAlrMt0vyQMkujmsRPfU3sCXXIODeXZTuPDcCR5 dLo7yAFxeKGvzVrXQyLjg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:sPATTXEHbrc=:cRQ/3M/S4ZUEvTjXnOnfy6 Zyso6en29+M6pIXw2OZShVS9Al0oMRWfPFhV29QYV346CzdIOVdILdsELHGrqbD6/N+iRirgu yoHgLJwBvtVsdxbHx2hxwPQatVWLFbHtawkisGoU76osiUVLlL5AVmOPFtA9WIZNhomblddq9 +TyVe9VZMl8JxjxwdPwAZvEjQkwDjT3p4K3qz1YCJQmP/R7L1khgN0w5ivFCnZTFGJKP16VVv GSuhvYscOGVXn/6MT0hNp8jiUAdwfFw3s/LfdE0IsA8AnZBPKxHdMPnkUUKygMj1hoPJJh9Ov WDI/mwSQ6QMdR+3N5yNQ7jYQetXi2Nk183OGyCD8wiKJezviG08/nK6eoOJM5Qe8ArP5dPxl9 +KGBlFpH+KVnDt11GdsZBl0hydWyIsJcWTLkWo6Ec+e/Gzwp3HGfspxJEVn/aDtDW+z1HwaiW orfGUZ6p2xq8CJGK6r8NPiXhhLqdd0He/rTu/QeAXnuwU0pQ1yFCa4GjkxGH4+tJss7VA7NV1 +zyrRB8b2Ji3PHJA0paCLOMADl7aSR7A7aH99RAmZNVaOJU0w07xJVW4F9L7Ey6k/TlllrOjt wCASAP/VjH3LmW5fvvea+mKteFuP0/+B8HLMXSXSWgwwhFPdlRuOdgbE0hqWo5TYQz73UIa6F Ovhaaz6TwDmTJ4Z1WHAh0q/q+qMyAivzaLATLf4Mh7AbsIJXyTohFfkp8vASwRlBM6YdqytZO V5LmOZEgramF/CkU6ZSjr0pmrj11JCKpR7tVsaSJwDm/7IrNrvDp1FoNvAKj3WDWPpkG9fXV2 Y+v9/mUSpLWJewg/+uujJRNV+VewQPeitlnl8JyRmG7KfYxWxNxP1vsQwfKUlSEZYp9EpU/K6 /ZPWp5r+yui3GpQyn61w== X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_FONT_FACE_BAD 0.001 HTML font face is not a word HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust RCVD_IN_MSPIKE_H3 0.001 Good reputation (+3) RCVD_IN_MSPIKE_WL 0.001 Mailspike good senders SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [rrdcached.pid, key-----begin.com] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [pve-devel] avoidable writes of pmxcfs to /var/lib/pve-cluster/config.db ? X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2021 20:50:56 -0000 hello proxmox team, i found that pmxcfs process is quite "chatty" and one of the top disk-writers on our proxmox nodes. i had a closer look, because i was curious, why wearout of our samsung EVO is already at 4% .=C2=A0 as disk I/O of our vms is typically very low,= so we used lower end ssd for those maschines. it seems pmxcfs is constantly writing into config.db-wal at a >10 kB/s and >10 writes/s rate level, whereas i can only see few changes in config.= db from my rough calculation, these writes probably sum up to several hundreds of gigabytes of disk blocks and >100mio iops written in a year, which isn't "just nothing" for lower-end ssd=C2=A0 (small and cheap ssd's = may only have some tens of TBW lifetime). i know that it's recommended to use enterprise ssd for proxmox, but as they are expensive i also dislike if they get avoidable wearout on any of our systems. what makes me raise my eyebrowe is, that it seems that most of the data written to the sqlite db seems to be unchanged data, i.e. i don't see significant changes in config.db over time, (compared with sqldiff), whereas the write-ahead-log at config.db-wal has quite high "flow rate". I cannot decide if this really is a must have, but it looks that writing of (at least parts of)=C2=A0 the cluster runtime data (like rsa key information) is being done in a "just dump it all down into the database" way. this may make it easy at the implementation level and easy for the programmer. i would love to hear a comment on this finding . maybe there is will/room for optimisation to avoid unnecessary disk wearout, saving avoidable database write/workload (ok it's tiny workload)=C2=A0 , but thus probably also lower the risk of database corruption in particular problem situations like server crash or whatever. regards Roland Kletzing ps: sorry, if this may look pointy-headed or bean-counting from a non-involved and sorry for posting here, but i was unsure if bugzilla was better for this, especially because i could not select "corosync-pve" as component for the bug/rfe ticket.=C2=A0 this is an opensource project and typically, opensource is something getting closer look and better optimization , which often makes it superior. at least it should be allowed to ask "is this intentional ?". # strace -f -p $(pidof pmxcfs)=C2=A0 -s32768=C2=A0 2>&1|grep pwrite64 | pv= -br -i10 >/dev/null 1.93MiB [18.8KiB/s] ^C # fatrace -c |stdbuf -oL grep pmxcfs | stdbuf -oL pv -lr -i60 >/dev/null [11.5 /s] # cat config.db-wal |strings|grep "BEGIN RSA" |cut -b 1-50|sort |uniq -c =C2=A0=C2=A0=C2=A0 331 pve-www.key-----BEGIN RSA PRIVATE KEY----- # cat config.db-wal |strings|grep "hp-ml350" |cut -b 1-50|sort |uniq -c =C2=A0=C2=A0=C2=A0 114 hp-ml350 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGN # iotop -a Total DISK READ:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B/s = | Total DISK WRITE:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 71.79 K/s Current DISK READ:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B/s | Current = DISK WRITE:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B/s =C2=A0 TID=C2=A0 PRIO=C2=A0 USER=C2=A0=C2=A0=C2=A0=C2=A0 DISK READ DISK W= RITE>=C2=A0 SWAPIN=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 IO COMMAND 26962 be/4 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20.00 K=C2= =A0=C2=A0 1295.00 K=C2=A0 0.00 %=C2=A0 0.00 % pmxcfs [cfs_loop] =C2=A08425 be/4 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0 32.00 K=C2=A0 0.00 %=C2=A0 0.00 % [kworker/u16:1-poll_mpt2sas0_statu] 26992 be/4 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00= B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 8.00 K=C2=A0 0.00 %=C2=A0 0.00 % rrdcache= d -B -b /var/lib/rrdcached/db/ -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock =C2=A07832 be/4 www-data=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0=C2=A0= 1024.00 B=C2=A0 0.00 %=C2=A0 0.00 % pveproxy worker =C2=A0=C2=A0=C2=A0 1 be/4 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0 0.00 %=C2= =A0 0.00 % init =C2=A0=C2=A0=C2=A0 2 be/4 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0 0.00 %=C2= =A0 0.00 % [kthreadd] =C2=A0=C2=A0=C2=A0 3 be/0 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0 0.00 %=C2= =A0 0.00 % [rcu_gp] =C2=A0=C2=A0=C2=A0 4 be/0 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0 0.00 %=C2= =A0 0.00 % [rcu_par_gp] =C2=A0=C2=A0=C2=A0 6 be/0 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0 0.00 %=C2= =A0 0.00 % [kworker/0:0H-kblockd] =C2=A0=C2=A0=C2=A0 8 be/0 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0 0.00 %=C2= =A0 0.00 % [mm_percpu_wq] =C2=A0=C2=A0=C2=A0 9 be/4 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.00 B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.00 B=C2=A0 0.00 %=C2= =A0 0.00 % [ksoftirqd/0]