From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <s.reiter@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id 61C8369F97
 for <pbs-devel@lists.proxmox.com>; Tue, 11 Aug 2020 10:07:53 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id 573321AB76
 for <pbs-devel@lists.proxmox.com>; Tue, 11 Aug 2020 10:07:23 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
 [212.186.127.180])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS id E04971AB69
 for <pbs-devel@lists.proxmox.com>; Tue, 11 Aug 2020 10:07:21 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
 by proxmox-new.maurer-it.com (Proxmox) with ESMTP id A0CAE4457F
 for <pbs-devel@lists.proxmox.com>; Tue, 11 Aug 2020 10:07:21 +0200 (CEST)
To: =?UTF-8?Q?Fabian_Gr=c3=bcnbichler?= <f.gruenbichler@proxmox.com>,
 Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com>
References: <20200810112509.70129-1-f.gruenbichler@proxmox.com>
 <20200810112509.70129-2-f.gruenbichler@proxmox.com>
 <e6a4b229-0ceb-4c5a-31c1-7f2bcfc7c208@proxmox.com>
 <1597132127.pvdvl5iryz.astroid@nora.none>
From: Stefan Reiter <s.reiter@proxmox.com>
Message-ID: <db158ad4-7b90-86a2-8f02-0b4f63f15262@proxmox.com>
Date: Tue, 11 Aug 2020 10:07:20 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.9.0
MIME-Version: 1.0
In-Reply-To: <1597132127.pvdvl5iryz.astroid@nora.none>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results:  0
 AWL -0.074 Adjusted score from AWL reputation of From: address
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 NICE_REPLY_A           -0.001 Looks like a legit reply (A)
 RCVD_IN_DNSWL_MED        -2.3 Sender listed at https://www.dnswl.org/,
 medium trust
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
 URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
 information. [restore.rs]
Subject: Re: [pbs-devel] [PATCH proxmox-backup-qemu] adapt to chunk reader
 changes
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 11 Aug 2020 08:07:53 -0000

On 8/11/20 9:53 AM, Fabian Grünbichler wrote:
> On August 10, 2020 5:13 pm, Stefan Reiter wrote:
>> makes sense in general, one comment inline
>>
>> On 8/10/20 1:25 PM, Fabian Grünbichler wrote:
>>> for verification of chunk crypt mode <=> index crypt mode match
>>>
>>> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
>>> ---
>>> follows patch #4 in proxmox-backup
>>>
>>>    src/restore.rs | 30 +++++++++++++-----------------
>>>    1 file changed, 13 insertions(+), 17 deletions(-)
>>>
>>> diff --git a/src/restore.rs b/src/restore.rs
>>> index e43d040..9d8c4a2 100644
>>> --- a/src/restore.rs
>>> +++ b/src/restore.rs
>>> @@ -1,5 +1,4 @@
>>>    use std::sync::{Arc, Mutex};
>>> -use std::collections::HashMap;
>>>    use std::io::SeekFrom;
>>>    use std::convert::TryInto;
>>>    
>>> @@ -27,7 +26,6 @@ pub(crate) struct RestoreTask {
>>>        runtime: Arc<Runtime>,
>>>        crypt_config: Option<Arc<CryptConfig>>,
>>>        client: OnceCell<Arc<BackupReader>>,
>>> -    chunk_reader: OnceCell<RemoteChunkReader>,
>>>        manifest: OnceCell<Arc<BackupManifest>>,
>>>        image_registry: Arc<Mutex<Registry<ImageAccessInfo>>>,
>>>    }
>>> @@ -59,7 +57,6 @@ impl RestoreTask {
>>>                crypt_config,
>>>                client: OnceCell::new(),
>>>                manifest: OnceCell::new(),
>>> -            chunk_reader: OnceCell::new(),
>>>                image_registry: Arc::new(Mutex::new(Registry::<ImageAccessInfo>::new())),
>>>            })
>>>        }
>>> @@ -94,15 +91,6 @@ impl RestoreTask {
>>>                true
>>>            ).await?;
>>>    
>>> -        let chunk_reader = RemoteChunkReader::new(
>>> -            client.clone(),
>>> -            self.crypt_config.clone(),
>>> -            HashMap::with_capacity(0),
>>> -        );
>>> -
>>> -        self.chunk_reader.set(chunk_reader)
>>> -            .map_err(|_| format_err!("already connected!"))?;
>>> -
>>>            let (manifest, _) = client.download_manifest().await?;
>>>    
>>>            self.manifest.set(Arc::new(manifest))
>>> @@ -150,9 +138,12 @@ impl RestoreTask {
>>>    
>>>            let most_used = index.find_most_used_chunks(8);
>>>    
>>> +        let file_info = manifest.lookup_file_info(&archive_name)?;
>>> +
>>>            let mut chunk_reader = RemoteChunkReader::new(
>>>                client.clone(),
>>>                self.crypt_config.clone(),
>>> +            file_info.chunk_crypt_mode(),
>>>                most_used,
>>>            );
>>>    
>>> @@ -219,11 +210,6 @@ impl RestoreTask {
>>>                None => bail!("not connected"),
>>>            };
>>>    
>>> -        let chunk_reader = match self.chunk_reader.get() {
>>> -            Some(chunk_reader) => chunk_reader.clone(),
>>> -            None => bail!("not connected"),
>>> -        };
>>> -
>>>            let manifest = match self.manifest.get() {
>>>                Some(manifest) => manifest.clone(),
>>>                None => bail!("no manifest"),
>>> @@ -231,6 +217,16 @@ impl RestoreTask {
>>>    
>>>            let index = client.download_fixed_index(&manifest, &archive_name).await?;
>>>            let archive_size = index.index_bytes();
>>> +        let most_used = index.find_most_used_chunks(8);
>>
>> You change the cache hint from previously an empty HashMap to the 8
>> most-used chunks, is that intentional?
>>
>> I don't believe that really does anything for a fixed index, and the
>> AsyncIndexReader already caches the currently accessed chunk.
> 
> I made it do that same thing that the other RemoteChunkReader in
> restore.rs does ;)
> 
>> If anything, maybe add the zero digest here to keep that in cache, or
>> even better maybe add that to the RemoteChunkReader?
> 
> yeah, could do that. the restore itself already skips it to write zeroes
> directly without even asking the chunk reader, so not sure how much we
> gain by teaching the chunk reader to skip them as well..
> 

Well then other usages of the readers would gain that functionality too, 
like mapping backup images (both in VMs and soon on the host). But we 
can change this later anyway, not too important atm.

>>
>>> +
>>> +        let file_info = manifest.lookup_file_info(&archive_name)?;
>>> +
>>> +        let chunk_reader = RemoteChunkReader::new(
>>> +            client.clone(),
>>> +            self.crypt_config.clone(),
>>> +            file_info.chunk_crypt_mode(),
>>> +            most_used,
>>> +        );
>>>    
>>>            let reader = AsyncIndexReader::new(index, chunk_reader);
>>>    
>>>
>>