From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D10A66A62F for ; Sat, 23 Jan 2021 10:15:10 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BD52322CE3 for ; Sat, 23 Jan 2021 10:15:10 +0100 (CET) Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 7C77922CCE for ; Sat, 23 Jan 2021 10:15:09 +0100 (CET) Received: by mail-ed1-x530.google.com with SMTP id c2so9019389edr.11 for ; Sat, 23 Jan 2021 01:15:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=amwes/jgiDL89RmOnmIFVJRmHvdnYGsHubw7ZqL0y/o=; b=hZ8feaiEMEJRRr2bgP5Q7r8JxCzww6OjugQsCP3er7x9IvOVj4oXU1HPgoOfI7O2WG YRbNkBDZoXSfP5LJvBq1w8Jix5QwdRkQKg+vth1CuylNx5/y5sfUbg1OPZHiD4q80uuP bfsb7J4ezCLkGjbOcgm/sav/4xFLWtbxaqWZODRaEUV/iRvnpJwHmXaW78BE+v1XWm/c uiP+JvRb+7HXDWTuxZ52IF3ui9XZm3PTRAnckF07j1ccWLEzDcrV5KrI6+h1pzsk3EeI 9nph1wlLPj7AefFgzMaLS7OsYEw1SN536Gp2akPypQ1+/huE4EJDKpwnU+br7NZg/ja7 pn9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=amwes/jgiDL89RmOnmIFVJRmHvdnYGsHubw7ZqL0y/o=; b=jTp0x0nY3likR3KFUtN3wp+kZ8Nql0APgqyQEiovQUg1m4XAjYmQCV7xo+oxHNNDnG lpgC5g4DmtygmykF2zE1nYtvCPKq8WfWFZppVn/bF0nBoS3DpPSiEbOCMYzIrsApJE+r Ta8eD5aDOjaaLEJz1XKOkSYW1x7CyM11nhDlcJxOuP2qU9AJ0aKvGA34UXZYfubWKtrj /JpRUwScQLdSYAGweOj/86+ikWoAxCUf4Wt4WAamniEDElgC3CKPSVhsE88tY2kxZ8SW iyEYq7/TTBuWDQgDy76I3H3s54d7jwIBwimk9Mjaxe1fL51up/GlTGbxlFhs6LA4f0Mf yyoQ== X-Gm-Message-State: AOAM532oxqV3l6h1+2zhRMuEP8NtkUKldUZkXBmhjifVhJRWtTFQU4E7 n0HWeVVWS17AYStYIXggvMQHbq2u+jx8198m6iXEEVVC1AJ+Dcuwin4= X-Google-Smtp-Source: ABdhPJyS7pdV1cm+kMwwArBCmt0HfOUzc4wSJ0ipBKO4j80hOXUqviBOKd9eKVI2CZ0tuRhmpmiGGzZIzk07Rt7l7nk= X-Received: by 2002:a05:6402:ce:: with SMTP id i14mr230876edu.42.1611393302992; Sat, 23 Jan 2021 01:15:02 -0800 (PST) MIME-Version: 1.0 References: <20210119144235.54f7jofljgjqpbts@chazelas.org> In-Reply-To: <20210119144235.54f7jofljgjqpbts@chazelas.org> From: Bruce Wainer Date: Sat, 23 Jan 2021 04:14:52 -0500 Message-ID: To: Proxmox VE development discussion , stephane@chazelas.org X-SPAM-LEVEL: Spam detection results: 0 AWL 0.072 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [pve-devel] [PATCH] ZFS ARC size not taken into account by pvestatd or ksmtuned X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2021 09:15:10 -0000 Hello Stephane, I believe this change is very important and would fix an issue I'm having on some of my servers. However the Proxmox team seems to be particular about how they accept patches (which I don't blame them for, it just is what it is). The details are here: https://pve.proxmox.com/wiki/Developer_Documentation - but in general you need to submit a Contributor Agreement, and submit the patch in a way that makes it easy for them to apply it. Alternatively if you are not willing to go through the steps of submitting this yourself, we might be able to work things out so I can submit it. Please let me know. Sincerely, Bruce Wainer On Wed, Jan 20, 2021 at 8:44 AM Stephane Chazelas wrote: > [note that I'm not subscribed to the list] > > Hello, > > I've been meaning to send this years ago. Sorry for the delay. > > We've been maintaining the patch below on our servers for years > now (since 2015), even before ZFS was officially supported by > PVE. > > We had experienced VM balloons swelling and processes in VMs > running out of memory even though the host had tons of RAM. > > We had tracked that down to pvestatd reclaiming memory from the > VMs. pvestatd targets 80% memory utilisation (in terms of memory > that is not free and not in buffers or caches: (memtotal - > memfree - buffers - cached) / memtotal). > > The problem is that the ZFS ARC is tracked independendly (not as > part of "buffers" or "cached" above). > > The size of that ARC cache also adapts with memory pressure. But > here, since the autoballooning frees memory as soon as it's used > up by the ARC, the ARC size grows and grows while VMs access > their disk, and we've got plenty of wasted free memory that is > never used. > > So in the end, with an ARC allowed to grow up to half the RAM, > we end up in a situation where pvestatd in effect targets 30% > max memory utilisation (with 20% free or in buffers and 50% in > ARC). > > Something similar happens for KSM (memory page deduplication). > /usr/sbin/ksmtuned monitors memory utilisation (again > total-cached-buffers-free) against kvm process memory > allocation, and tells the ksm daemon to scan more and more > pages, more and more aggressively as long as the "used" memory > is above 80%. > > That probably explains why performances decrease significantly > after a while and why doing a "echo 3 > > /proc/sys/vm/drop_caches" (which clears buffers, caches *AND* > the ZFS arc cache) gives a second life to the system. > > (by the way, a recent version of ProcFSTools.pm added a > read_pressure function, but it doesn't look like it's used > anywhere). > > --- /usr/share/perl5/PVE/ProcFSTools.pm.distrib 2020-12-03 > 15:53:17.000000000 +0000 > +++ /usr/share/perl5/PVE/ProcFSTools.pm 2021-01-19 13:44:42.480272044 +0000 > @@ -268,6 +268,19 @@ sub read_meminfo { > > $res->{memtotal} = $d->{memtotal}; > $res->{memfree} = $d->{memfree} + $d->{buffers} + $d->{cached}; > + > + # Add the ZFS ARC if any > + if (my $fh_arc = IO::File->new("/proc/spl/kstat/zfs/arcstats", "r")) { > + while (my $line = <$fh_arc>) { > + if ($line =~ m/^size .* (\d+)/) { > + # "size" already in bytes > + $res->{memfree} += $1; > + last; > + } > + } > + close($fh_arc); > + } > + > $res->{memused} = $res->{memtotal} - $res->{memfree}; > > $res->{swaptotal} = $d->{swaptotal}; > --- /usr/sbin/ksmtuned.distrib 2020-07-24 10:04:45.827828719 +0100 > +++ /usr/sbin/ksmtuned 2021-01-19 14:37:43.416360037 +0000 > @@ -75,10 +75,17 @@ committed_memory () { > ps -C "$progname" -o vsz= | awk '{ sum += $1 }; END { print sum }' > } > > -free_memory () { > - awk '/^(MemFree|Buffers|Cached):/ {free += $2}; END {print free}' \ > - /proc/meminfo > -} > +free_memory () ( > + shopt -s nullglob > + exec awk ' > + NR == FNR { > + if (/^(MemFree|Buffers|Cached):/) free += $2 > + next > + } > + $1 == "size" {free += int($3/1024)} > + END {print free} > + ' /proc/meminfo /proc/spl/kstat/zfs/[a]rcstats > +) > > increase_npages() { > local delta > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > >