From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 90B1970275; Mon, 7 Jun 2021 10:04:20 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7C610E285; Mon, 7 Jun 2021 10:03:50 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 618FAE276; Mon, 7 Jun 2021 10:03:45 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 3665B42D82; Mon, 7 Jun 2021 10:03:45 +0200 (CEST) To: Wolfgang Bumiller Cc: pve-devel@lists.proxmox.com, pbs-devel@lists.proxmox.com References: <20210602143833.4423-1-s.reiter@proxmox.com> <20210602143833.4423-9-s.reiter@proxmox.com> <20210604121627.hrvzcoju6nape6cf@olga.proxmox.com> From: Stefan Reiter Message-ID: <6bb3dd5a-0dcd-5183-0bb9-4ecb9d6f0e4d@proxmox.com> Date: Mon, 7 Jun 2021 10:03:44 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20210604121627.hrvzcoju6nape6cf@olga.proxmox.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.999 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [pbs-devel] [PATCH proxmox-backup-qemu 8/9] add shared_cache module X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jun 2021 08:04:20 -0000 On 6/4/21 2:16 PM, Wolfgang Bumiller wrote: > On Wed, Jun 02, 2021 at 04:38:32PM +0200, Stefan Reiter wrote: >> Provides a shared AsyncLruCache of 256MB (w/ 4MB chunks) that can be >> used by multiple readers at the same time. It is dropped once no more >> readers exist, so the memory gets freed if all QEMU block/pbs instances >> disappear. >> >> Signed-off-by: Stefan Reiter >> --- >> src/lib.rs | 7 ++++++- >> src/shared_cache.rs | 36 ++++++++++++++++++++++++++++++++++++ >> 2 files changed, 42 insertions(+), 1 deletion(-) >> create mode 100644 src/shared_cache.rs >> >> diff --git a/src/lib.rs b/src/lib.rs >> index 05d7b58..aa167f7 100644 >> --- a/src/lib.rs >> +++ b/src/lib.rs >> @@ -25,6 +25,7 @@ mod restore; >> use restore::*; >> >> mod tools; >> +mod shared_cache; >> >> pub const PROXMOX_BACKUP_DEFAULT_CHUNK_SIZE: u64 = 1024*1024*4; >> >> @@ -804,7 +805,11 @@ pub extern "C" fn proxmox_restore_connect_async( >> pub extern "C" fn proxmox_restore_disconnect(handle: *mut ProxmoxRestoreHandle) { >> >> let restore_task = handle as * mut Arc; >> - unsafe { Box::from_raw(restore_task) }; //drop(restore_task) >> + let restore_task = unsafe { Box::from_raw(restore_task) }; >> + drop(restore_task); >> + >> + // after dropping, cache may be unused (if no other handles open) >> + shared_cache::shared_chunk_cache_cleanup(); >> } >> >> /// Restore an image (sync) >> diff --git a/src/shared_cache.rs b/src/shared_cache.rs >> new file mode 100644 >> index 0000000..bebae5b >> --- /dev/null >> +++ b/src/shared_cache.rs >> @@ -0,0 +1,36 @@ >> +use once_cell::sync::OnceCell; >> +use proxmox_backup::backup::ChunkCache; >> +use proxmox_backup::tools::async_lru_cache::AsyncLruCache; >> +use std::sync::{Arc, Mutex}; >> + >> +const SHARED_CACHE_CAPACITY: usize = 64; // 256 MB >> +static SHARED_CACHE: OnceCell>> = OnceCell::new(); > > OnceCell *and* Option is a bit too much, `get_shared_chunk_chache()` can > just initialize it in `get_or_init()` directly. > Then how do we drop the ChunkCache? I used OnceCell since std::sync::Mutex doesn't seem to have a 'const new', not because of 'Option' semantics. >> + >> +pub fn get_shared_chunk_cache() -> ChunkCache { >> + let mut guard = SHARED_CACHE >> + .get_or_init(|| Mutex::new(None)) >> + .lock() >> + .unwrap(); >> + match &*guard { >> + Some(cache) => Arc::clone(cache), >> + None => { >> + let cache = Arc::new(AsyncLruCache::new(SHARED_CACHE_CAPACITY)); >> + *guard = Some(Arc::clone(&cache)); >> + cache >> + } >> + } >> +} >> + >> +pub fn shared_chunk_cache_cleanup() { >> + let mut guard = SHARED_CACHE >> + .get_or_init(|| Mutex::new(None)) >> + .lock() >> + .unwrap(); >> + if let Some(arc) = guard.as_ref() { >> + let refcount = Arc::strong_count(arc); >> + if refcount == 1 { >> + // no one else is using the cache anymore, drop it >> + let _drop = guard.take(); >> + } >> + } >> +} >> -- >> 2.30.2