From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id F32351FF139 for ; Tue, 24 Feb 2026 09:46:35 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C0E691E53F; Tue, 24 Feb 2026 09:47:26 +0100 (CET) From: Markus Ebner To: pve-devel@lists.proxmox.com Subject: [PATCH container] close #7342: Extend qga file-read with chunked access for large files Date: Mon, 23 Feb 2026 21:16:47 +0100 Message-ID: <20260223201648.297620-2-info@ebner-markus.de> In-Reply-To: <20260223201648.297620-1-info@ebner-markus.de> References: <20260223201648.297620-1-info@ebner-markus.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.712 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_PASS -0.1 DMARC pass policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.798 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.79 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.547 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record X-MailFrom: info@ebner-markus.de X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation Message-ID-Hash: JQHHY36KLWRU5DQZKIE2PAWUJQC2BOFA X-Message-ID-Hash: JQHHY36KLWRU5DQZKIE2PAWUJQC2BOFA X-Mailman-Approved-At: Tue, 24 Feb 2026 09:47:29 +0100 CC: Markus Ebner X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The file-read command of the QEMU guest agent previously had several practical limitations. It always read a fixed 16 MiB block starting at offset 0, making it impossible to retrieve larger files in multiple chunks. On busy or resource‑constrained hosts, requests for large files often timed out because the agent attempted to read and JSON‑encode the entire 16 MiB block at once. Binary data was also returned as raw JSON strings with extensive escaping, which inflated payload size and caused compatibility issues with some JSON parsers. This patch extends the file-read method with three new parameters: - decode — Controls whether the base64‑encoded data returned by the guest agent should be decoded before being sent back through the API. When disabled, the base64 string is passed through unchanged, which is ideal for binary data and mirrors the existing encode parameter of file-write. - offset — Allows reading from an arbitrary byte offset within the file. - count — Allows requesting a smaller number of bytes than the internal 16 MiB limit, avoiding unnecessary overhead and reducing timeout risk. With these additions, the behavior now mirrors standard file operations (fopen, fseek, fread). Reading beyond EOF returns zero bytes. Seek can choose any non-negative position within the file, without bounds checking. Reading out of bounds returns 0 bytes. This allows conveniently reading an entire file in a robust way: while(truncated && content.length != 0) {} and also enables things like tailing a changing file. This makes the file-read command significantly more flexible. All parameter additions were done in a backwards-compatible fashion. Signed-off-by: Markus Ebner --- src/PVE/API2/Qemu/Agent.pm | 54 ++++++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 8 deletions(-) diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm index de36ce1e..ccd1dca2 100644 --- a/src/PVE/API2/Qemu/Agent.pm +++ b/src/PVE/API2/Qemu/Agent.pm @@ -464,6 +464,28 @@ __PACKAGE__->register_method({ 'pve-vmid', { completion => \&PVE::QemuServer::complete_vmid_running }, ), + decode => { + type => 'boolean', + optional => 1, + default => 1, + description => + "Data received from the QEMU Guest-Agent is base64 encoded. If this is set to true, the data is decoded." + . "Otherwise the content is forwarded with base64 encoding - defaults to true.", + }, + offset => { + type => 'integer', + optional => 1, + default => 0, + description => "Offset to start reading at", + }, + count => { + type => 'integer', + optional => 1, + minimum => 0, + maximum => $MAX_READ_SIZE, + default => $MAX_READ_SIZE, + description => "Number of bytes to read.", + }, file => { type => 'string', description => 'The path to the file', @@ -487,6 +509,9 @@ __PACKAGE__->register_method({ }, code => sub { my ($param) = @_; + my $param_offset = int($param->{offset} // 0); + my $param_decode = $param->{decode} // 1; + my $param_count = int($param->{count} // $MAX_READ_SIZE); my $vmid = $param->{vmid}; my $conf = PVE::QemuConfig->load_config($vmid); @@ -494,18 +519,33 @@ __PACKAGE__->register_method({ my $qgafh = agent_cmd($vmid, $conf, "file-open", { path => $param->{file} }, "can't open file"); - my $bytes_left = $MAX_READ_SIZE; + if ($param_offset > 0) { + my $seek = mon_cmd( + $vmid, "guest-file-seek", + handle => $qgafh, + offset => $param_offset, + whence => 'set', + ); + check_agent_error($seek, "can't seek to offset position"); + } + + my $bytes_read = 0; my $eof = 0; my $read_size = 1024 * 1024; my $content = ""; - while ($bytes_left > 0 && !$eof) { + while ($bytes_read < $param_count && !$eof) { + my $bytes_left = $param_count - $bytes_read; + my $chunk_size = $bytes_left < $read_size ? $bytes_left : $read_size; my $read = - mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($read_size)); + mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($chunk_size)); check_agent_error($read, "can't read from file"); - $content .= decode_base64($read->{'buf-b64'}); - $bytes_left -= $read->{count}; + my $chunk = $read->{'buf-b64'}; + $chunk = decode_base64($chunk) if $param_decode; + $content .= $chunk; + + $bytes_read += $read->{count}; $eof = $read->{eof} // 0; } @@ -514,12 +554,10 @@ __PACKAGE__->register_method({ my $result = { content => $content, - 'bytes-read' => ($MAX_READ_SIZE - $bytes_left), + 'bytes-read' => $bytes_read, }; if (!$eof) { - warn - "agent file-read: reached maximum read size: $MAX_READ_SIZE bytes. output might be truncated.\n"; $result->{truncated} = 1; } -- 2.53.0