[PATCH qemu-server v3 0/3] Extend qga file-read with chunked access for large files

[PATCH qemu-server v3 0/3] Extend qga file-read with chunked access for large files

[Markus Ebner]

The file-read command of the QEMU guest agent previously had several
practical limitations. It always read a fixed 16 MiB block starting at
offset 0, making it impossible to retrieve larger files in multiple
chunks. On busy or resource‑constrained hosts, requests for large files
often timed out because the agent attempted to read and JSON‑encode the
entire 16 MiB block at once (while producing high CPU load in both
qemu-guest-agent, as well as pvedaemon).

Since decode wasn't optional, readinb binary data encoded as JSON
string regularly caused highly inflated payload sizes due to extensive
unicode escaping.

This series extends the file-read method with three new parameters:

- decode — Controls whether the base64‑encoded data returned by the
  guest agent should be decoded before being sent back through the API.
  When disabled, the base64 string is passed through unchanged, which is
  ideal for binary data and mirrors the existing encode parameter of
  file-write.

- offset — Allows reading from an arbitrary byte offset within the
  file.

- count — Allows requesting a smaller number of bytes than the
  internal 16 MiB limit, granting more flexibility to avoid API
  timeouts.

With these additions, the agent/file-read API now essentially behaves
like a pread() systemcall.
All parameter additions were done in a backwards-compatible fashion.

Markus Ebner (3):
  agent: file-read: Allow specifying max number of bytes to read
  agent: file-read: Allow maintaining base64-encoding of content
  agent: file-read: Allow specifying byte offset to start reading at

 src/PVE/API2/Qemu/Agent.pm | 62 ++++++++++++++++++++++++++++++++------
 1 file changed, 53 insertions(+), 9 deletions(-)

-- 
2.53.0

[PATCH qemu-server v3 1/3] agent: file-read: Allow specifying max number of bytes to read

[Markus Ebner]

The previous file-read implementation in the Proxmox API always
attempted to read a fixed 16 MiB. On busy or resource-constrainted
hosts, that often caused request timeouts, with no option to
reduce the number of read bytes to at least read something.

The new parameter count now allows specifying the exact number of
bytes that should be read from the given file. To be backwards
compatible with the previous behavior, it defaults to 16 MiB.
These 16 MiB are further also the maximum allowed value.

Signed-off-by: Markus Ebner <info@ebner-markus.de>
---
 src/PVE/API2/Qemu/Agent.pm | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm
index de36ce1e..c6155f02 100644
--- a/src/PVE/API2/Qemu/Agent.pm
+++ b/src/PVE/API2/Qemu/Agent.pm
@@ -464,6 +464,14 @@ __PACKAGE__->register_method({
                 'pve-vmid',
                 { completion => \&PVE::QemuServer::complete_vmid_running },
             ),
+            count => {
+                type => 'integer',
+                optional => 1,
+                minimum => 1,
+                maximum => $MAX_READ_SIZE,
+                default => $MAX_READ_SIZE,
+                description => "Number of bytes to read.",
+            },
             file => {
                 type => 'string',
                 description => 'The path to the file',
@@ -481,12 +489,13 @@ __PACKAGE__->register_method({
             truncated => {
                 type => 'boolean',
                 optional => 1,
-                description => "If set to 1, the output is truncated and not complete",
+                description => "If set to 1, the read did not reach the end of the file.",
             },
         },
     },
     code => sub {
         my ($param) = @_;
+        my $count = $param->{count} // $MAX_READ_SIZE;
 
         my $vmid = $param->{vmid};
         my $conf = PVE::QemuConfig->load_config($vmid);
@@ -494,18 +503,20 @@ __PACKAGE__->register_method({
         my $qgafh =
             agent_cmd($vmid, $conf, "file-open", { path => $param->{file} }, "can't open file");
 
-        my $bytes_left = $MAX_READ_SIZE;
+        my $bytes_read = 0;
         my $eof = 0;
         my $read_size = 1024 * 1024;
         my $content = "";
 
-        while ($bytes_left > 0 && !$eof) {
+        while ($bytes_read < $count && !$eof) {
+            my $bytes_left = $count - $bytes_read;
+            my $chunk_size = $bytes_left < $read_size ? $bytes_left : $read_size;
             my $read =
-                mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($read_size));
+                mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($chunk_size));
             check_agent_error($read, "can't read from file");
 
             $content .= decode_base64($read->{'buf-b64'});
-            $bytes_left -= $read->{count};
+            $bytes_read += $read->{count};
             $eof = $read->{eof} // 0;
         }
 
@@ -514,12 +525,14 @@ __PACKAGE__->register_method({
 
         my $result = {
             content => $content,
-            'bytes-read' => ($MAX_READ_SIZE - $bytes_left),
+            'bytes-read' => $bytes_read,
         };
 
         if (!$eof) {
-            warn
-                "agent file-read: reached maximum read size: $MAX_READ_SIZE bytes. output might be truncated.\n";
+            if (!defined($param->{count})) {
+                warn "agent file-read: reached maximum read size: $MAX_READ_SIZE bytes."
+                    . " output might be truncated.\n";
+            }
             $result->{truncated} = 1;
         }
 
-- 
2.53.0

[PATCH qemu-server v3 2/3] agent: file-read: Allow maintaining base64-encoding of content

[Markus Ebner]

The previous implementation always decoded the base64-encoded content
received from the qemu-guest-agent file-read call. Since JSON strings
must be comliant unicode text, binary data got escaped using unicode
escape sequences, using the pattern: \u00XX - with XX being the hex
value of the byte to encode. For certain binary files, this lead to a
massively inflated payload size of the API response.

Comparison on my test system:
For a 4MiB test-file generated using
dd if=/dev/urandom bs=4M count=1

- Reading it with decode=1 transfers 8.61MiB and takes 5700ms on avg.
- Reading it with decode=0 transfers 5.59MiB and takes 3300ms on avg.

To be backwards compatible, the decode parameter defaults to 1.

Signed-off-by: Markus Ebner <info@ebner-markus.de>
---
 src/PVE/API2/Qemu/Agent.pm | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm
index c6155f02..49dbef09 100644
--- a/src/PVE/API2/Qemu/Agent.pm
+++ b/src/PVE/API2/Qemu/Agent.pm
@@ -472,6 +472,15 @@ __PACKAGE__->register_method({
                 default => $MAX_READ_SIZE,
                 description => "Number of bytes to read.",
             },
+            decode => {
+                type => 'boolean',
+                optional => 1,
+                default => 1,
+                description => "Data received from the QEMU Guest-Agent is base64 encoded."
+                    . " If this is set to true, the data is decoded."
+                    . " Otherwise the content is forwarded with base64 encoding."
+                    . " Defaults to true.",
+            },
             file => {
                 type => 'string',
                 description => 'The path to the file',
@@ -496,6 +505,7 @@ __PACKAGE__->register_method({
     code => sub {
         my ($param) = @_;
         my $count = $param->{count} // $MAX_READ_SIZE;
+        my $decode = $param->{decode} // 1;
 
         my $vmid = $param->{vmid};
         my $conf = PVE::QemuConfig->load_config($vmid);
@@ -515,7 +525,10 @@ __PACKAGE__->register_method({
                 mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($chunk_size));
             check_agent_error($read, "can't read from file");
 
-            $content .= decode_base64($read->{'buf-b64'});
+            my $chunk = $read->{'buf-b64'};
+            $chunk = decode_base64($chunk) if $decode;
+            $content .= $chunk;
+
             $bytes_read += $read->{count};
             $eof = $read->{eof} // 0;
         }
-- 
2.53.0

[PATCH qemu-server v3 3/3] agent: file-read: Allow specifying byte offset to start reading at

[Markus Ebner]

The previous implementation only allowed reads of fixed size starting
at byte offset 0. This made the Proxmox's agent/file-read unsuitable
for reading large files.
The new offset parameter allows specifying a byte offset position
relative to the start of the file (offset 0) from which the following
read should be performed. The implementation's behavior follows the
lseek semantics - directly following qemu-guest-agent's behavior:

- fseek() does not perform validation against the file size
- Seeking beyond the end of the file is not an error but explicitly
  allowed and specified behavior
- Subsequent fread() calls at positions beyond the file size return
  zero bytes

Additionally, this is equivalent to the behavior of POSIX pread().
Reading a large file block-by-block thus becomes (pseudo-code):
let content = [];
while(response.truncated) {
    response = proxmox.agent_file_read(offset = content.length);
    content .= response.content;
}

To be backwards compatible, offset defaults to 0.

Signed-off-by: Markus Ebner <info@ebner-markus.de>
---
 src/PVE/API2/Qemu/Agent.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm
index 49dbef09..a4234968 100644
--- a/src/PVE/API2/Qemu/Agent.pm
+++ b/src/PVE/API2/Qemu/Agent.pm
@@ -485,6 +485,13 @@ __PACKAGE__->register_method({
                 type => 'string',
                 description => 'The path to the file',
             },
+            offset => {
+                type => 'integer',
+                optional => 1,
+                minimum => 0,
+                default => 0,
+                description => "Offset to start reading at",
+            },
         },
     },
     returns => {
@@ -506,6 +513,7 @@ __PACKAGE__->register_method({
         my ($param) = @_;
         my $count = $param->{count} // $MAX_READ_SIZE;
         my $decode = $param->{decode} // 1;
+        my $offset = $param->{offset} // 0;
 
         my $vmid = $param->{vmid};
         my $conf = PVE::QemuConfig->load_config($vmid);
@@ -513,6 +521,16 @@ __PACKAGE__->register_method({
         my $qgafh =
             agent_cmd($vmid, $conf, "file-open", { path => $param->{file} }, "can't open file");
 
+        if ($offset > 0) {
+            my $seek = mon_cmd(
+                $vmid, "guest-file-seek",
+                handle => $qgafh,
+                offset => int($offset),
+                whence => 'set',
+            );
+            check_agent_error($seek, "can't seek to offset position");
+        }
+
         my $bytes_read = 0;
         my $eof = 0;
         my $read_size = 1024 * 1024;
-- 
2.53.0

applied: [PATCH qemu-server v3 0/3] Extend qga file-read with chunked access for large files

[Fiona Ebner]

On Thu, 26 Feb 2026 13:31:16 +0100, Markus Ebner wrote:
> The file-read command of the QEMU guest agent previously had several
> practical limitations. It always read a fixed 16 MiB block starting at
> offset 0, making it impossible to retrieve larger files in multiple
> chunks. On busy or resource‑constrained hosts, requests for large files
> often timed out because the agent attempted to read and JSON‑encode the
> entire 16 MiB block at once (while producing high CPU load in both
> qemu-guest-agent, as well as pvedaemon).
> 
> [...]

Applied, thanks!

[1/3] agent: file-read: Allow specifying max number of bytes to read
      commit: 5c623f82c07624528c6b107cc6c9660ec6b84d98
[2/3] agent: file-read: Allow maintaining base64-encoding of content
      commit: a1397a11dd6092ae252d2cc8c65b1ca1eb815e58
[3/3] agent: file-read: Allow specifying byte offset to start reading at
      commit: ec268ede5b1cfc050ff3a6d252bebabacee3154e