From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DDE3EB6DB; Thu, 24 Nov 2022 11:26:19 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C12542CAE8; Thu, 24 Nov 2022 11:26:19 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS; Thu, 24 Nov 2022 11:26:18 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 44A04445E8; Thu, 24 Nov 2022 11:26:18 +0100 (CET) Date: Thu, 24 Nov 2022 11:26:17 +0100 From: Wolfgang Bumiller To: Daniel Tschlatscher Cc: pve-devel@lists.proxmox.com, pbs-devel@lists.proxmox.com, pmg-devel@lists.proxmox.com Message-ID: <20221124102617.6bdrqcrjj5zyhi6o@casey.proxmox.com> References: <20221123145210.592981-1-d.tschlatscher@proxmox.com> <20221123145210.592981-2-d.tschlatscher@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221123145210.592981-2-d.tschlatscher@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.228 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [tasks.rs] Subject: [pmg-devel] applied: [pbs-devel] [PATCH backup v4] make tasklog downloadable in the backup server backend X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2022 10:26:19 -0000 applied with a cleanup followup commit, see below On Wed, Nov 23, 2022 at 03:52:07PM +0100, Daniel Tschlatscher wrote: > The read_tasklog API call now stream the whole log file if the query > parameter 'download' is set to true. If the limit parameter is set to > 0, all lines in the tasklog will be returned in json format. > > To make a file stream and a json response in the same API call work, I > had to use one of the lower level apimethod types from the > proxmox-router. Therefore, the routing declarations and parameter > schemas have been changed accordingly. > > Signed-off-by: Daniel Tschlatscher > --- > I sent this separately, as the other 3 patches in this version do not > concern the backup patch in any way. > > Changes from v3: > * API parameter 'download' for streaming the file directly > * option 'limit=0' returns the whole task log > * read_task_log_lines() function is inline in the code again as it's > the only occurrence and doesn't add that much readability > > src/api2/node/tasks.rs | 209 +++++++++++++++++++++++++---------------- > 1 file changed, 129 insertions(+), 80 deletions(-) > > diff --git a/src/api2/node/tasks.rs b/src/api2/node/tasks.rs > index cbd0894e..258c3e86 100644 > --- a/src/api2/node/tasks.rs > +++ b/src/api2/node/tasks.rs > @@ -2,10 +2,18 @@ use std::fs::File; > use std::io::{BufRead, BufReader}; > > use anyhow::{bail, Error}; > +use futures::FutureExt; > +use http::request::Parts; > +use http::{header, Response, StatusCode}; > +use hyper::Body; > +use proxmox_async::stream::AsyncReaderStream; > use serde_json::{json, Value}; > > -use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap}; > -use proxmox_schema::api; > +use proxmox_router::{ > + list_subdirs_api_method, ApiHandler, ApiMethod, ApiResponseFuture, Permission, Router, > + RpcEnvironment, SubdirMap, > +}; > +use proxmox_schema::{api, BooleanSchema, IntegerSchema, ObjectSchema, Schema}; > use proxmox_sys::sortable; > > use pbs_api_types::{ > @@ -19,6 +27,27 @@ use crate::api2::pull::check_pull_privs; > use pbs_config::CachedUserInfo; > use proxmox_rest_server::{upid_log_path, upid_read_status, TaskListInfoIterator, TaskState}; > > +pub const START_PARAM_SCHEMA: Schema = > + IntegerSchema::new("Start at this line when reading the tasklog") > + .minimum(0) > + .default(0) > + .schema(); > + > +pub const LIMIT_PARAM_SCHEMA: Schema = > + IntegerSchema::new("The amount of lines to read from the tasklog. Setting this parameter to 0 will return all lines until the end of the file.") ^ Please split lines longer than 100 chars, like: IntegerSchema::new( "The amount of lines to read from the tasklog. \ Setting this parameter to 0 will return all lines until the end of the file." ) (...) > + if download { > + if !param["start"].is_null() > + || !param["limit"].is_null() > + || !param["test-status"].is_null() > + { > + bail!("Parameter 'download' cannot be used with other parameters"); > + } > > - for line in BufReader::new(file).lines() { > - match line { > - Ok(line) => { > - count += 1; > - if count < start { > - continue; > - }; > - if limit == 0 { > - continue; > - }; > + let header_disp = format!("attachment; filename={}", &upid.to_string()); > + let stream = AsyncReaderStream::new(tokio::fs::File::open(path).await?); > + > + Ok(Response::builder() > + .status(StatusCode::OK) > + .header(header::CONTENT_TYPE, "text/plain") > + .header(header::CONTENT_DISPOSITION, &header_disp) > + .body(Body::wrap_stream(stream)) > + .unwrap()) ^ This could just have a `return` in front of it, so the remaining code isn't that much indented. > + } else { > + let start = param["start"].as_u64().unwrap_or(0); > + let mut limit = param["limit"].as_u64().unwrap_or(50); > + let test_status = param["test-status"].as_bool().unwrap_or(false); > + > + let file = File::open(path)?; > + > + let mut count: u64 = 0; > + let mut lines: Vec = vec![]; > + let read_until_end = if limit == 0 { true } else { false }; ^ booleans are already booleans, please avoid this construct in the future. clippy can catch this btw.