[PATCH qemu-server v2 0/3] Extend qga file-read with chunked access for large files

[PATCH qemu-server v2 0/3] Extend qga file-read with chunked access for large files

[Markus Ebner]

The file-read command of the QEMU guest agent previously had several
practical limitations. It always read a fixed 16 MiB block starting at
offset 0, making it impossible to retrieve larger files in multiple
chunks. On busy or resource‑constrained hosts, requests for large files
often timed out because the agent attempted to read and JSON‑encode the
entire 16 MiB block at once (while producing high CPU load in both
qemu-guest-agent, as well as pvedaemon).

Since decode wasn't optional, readinb binary data encoded as JSON
string regularly caused highly inflated payload sizes due to extensive
unicode escaping.

This series extends the file-read method with three new parameters:

- decode — Controls whether the base64‑encoded data returned by the
  guest agent should be decoded before being sent back through the API.
  When disabled, the base64 string is passed through unchanged, which is
  ideal for binary data and mirrors the existing encode parameter of
  file-write.

- offset — Allows reading from an arbitrary byte offset within the
  file.

- count — Allows requesting a smaller number of bytes than the
  internal 16 MiB limit, granting more flexibility to avoid API
  timeouts.

With these additions, the agent/file-read API now essentially behaves
like a pread() systemcall.
All parameter additions were done in a backwards-compatible fashion.

Markus Ebner (3):
  agent: file-read: Allow specifying max number of bytes to read
  agent: file-read: Allow maintaining base64-encoding of content
  agent: file-read: Allow specifying byte offset to start reading at

 src/PVE/API2/Qemu/Agent.pm | 60 +++++++++++++++++++++++++++++++++-----
 1 file changed, 52 insertions(+), 8 deletions(-)

-- 
2.53.0

[PATCH qemu-server v2 1/3] agent: file-read: Allow specifying max number of bytes to read

[Markus Ebner]

The previous file-read implementation in the Proxmox API always
attempted to read a fixed 16 MiB. On busy or resource-constrainted
hosts, that often caused request timeouts, with no option to
reduce the number of read bytes to at least read something.

The new parameter count now allows specifying the exact number of
bytes that should be read from the given file. To be backwards
compatible with the previous behavior, it defaults to 16 MiB.
These 16 MiB are further also the maximum allowed value.

Signed-off-by: Markus Ebner <info@ebner-markus.de>
---
 src/PVE/API2/Qemu/Agent.pm | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm
index de36ce1e..f32e1dc2 100644
--- a/src/PVE/API2/Qemu/Agent.pm
+++ b/src/PVE/API2/Qemu/Agent.pm
@@ -464,6 +464,14 @@ __PACKAGE__->register_method({
                 'pve-vmid',
                 { completion => \&PVE::QemuServer::complete_vmid_running },
             ),
+            count => {
+                type => 'integer',
+                optional => 1,
+                minimum => 0,
+                maximum => $MAX_READ_SIZE,
+                default => $MAX_READ_SIZE,
+                description => "Number of bytes to read.",
+            },
             file => {
                 type => 'string',
                 description => 'The path to the file',
@@ -487,6 +495,7 @@ __PACKAGE__->register_method({
     },
     code => sub {
         my ($param) = @_;
+        my $count = int($param->{count} // $MAX_READ_SIZE);
 
         my $vmid = $param->{vmid};
         my $conf = PVE::QemuConfig->load_config($vmid);
@@ -494,18 +503,20 @@ __PACKAGE__->register_method({
         my $qgafh =
             agent_cmd($vmid, $conf, "file-open", { path => $param->{file} }, "can't open file");
 
-        my $bytes_left = $MAX_READ_SIZE;
+        my $bytes_read = 0;
         my $eof = 0;
         my $read_size = 1024 * 1024;
         my $content = "";
 
-        while ($bytes_left > 0 && !$eof) {
+        while ($bytes_read < $count && !$eof) {
+            my $bytes_left = $count - $bytes_read;
+            my $chunk_size = $bytes_left < $read_size ? $bytes_left : $read_size;
             my $read =
-                mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($read_size));
+                mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($chunk_size));
             check_agent_error($read, "can't read from file");
 
             $content .= decode_base64($read->{'buf-b64'});
-            $bytes_left -= $read->{count};
+            $bytes_read += $read->{count};
             $eof = $read->{eof} // 0;
         }
 
@@ -514,12 +525,14 @@ __PACKAGE__->register_method({
 
         my $result = {
             content => $content,
-            'bytes-read' => ($MAX_READ_SIZE - $bytes_left),
+            'bytes-read' => $bytes_read,
         };
 
         if (!$eof) {
-            warn
-                "agent file-read: reached maximum read size: $MAX_READ_SIZE bytes. output might be truncated.\n";
+            if (!defined $param->{count}) {
+                warn "agent file-read: reached maximum read size: $MAX_READ_SIZE bytes."
+                    . " output might be truncated.\n";
+            }
             $result->{truncated} = 1;
         }
 
-- 
2.53.0

[PATCH qemu-server v2 2/3] agent: file-read: Allow maintaining base64-encoding of content

[Markus Ebner]

The previous implementation always decoded the base64-encoded content
received from the qemu-guest-agent file-read call. Since JSON strings
must be comliant unicode text, binary data got escaped using unicode
escape sequences, using the pattern: \u00XX - with XX being the hex
value of the byte to encode. For certain binary files, this lead to a
massively inflated payload size of the API response.

Comparison on my test system:
For a 4MiB test-file generated using
dd if=/dev/urandom bs=4M count=1

- Reading it with decode=1 transfers 8.61MiB and takes 5700ms on avg.
- Reading it with decode=0 transfers 5.59MiB and takes 3300ms on avg.

To be backwards compatible, the decode parameter defaults to 1.

Signed-off-by: Markus Ebner <info@ebner-markus.de>
---
 src/PVE/API2/Qemu/Agent.pm | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm
index f32e1dc2..1f5962e9 100644
--- a/src/PVE/API2/Qemu/Agent.pm
+++ b/src/PVE/API2/Qemu/Agent.pm
@@ -472,6 +472,15 @@ __PACKAGE__->register_method({
                 default => $MAX_READ_SIZE,
                 description => "Number of bytes to read.",
             },
+            decode => {
+                type => 'boolean',
+                optional => 1,
+                default => 1,
+                description => "Data received from the QEMU Guest-Agent is base64 encoded."
+                    . " If this is set to true, the data is decoded."
+                    . " Otherwise the content is forwarded with base64 encoding."
+                    . " Defaults to true.",
+            },
             file => {
                 type => 'string',
                 description => 'The path to the file',
@@ -496,6 +505,7 @@ __PACKAGE__->register_method({
     code => sub {
         my ($param) = @_;
         my $count = int($param->{count} // $MAX_READ_SIZE);
+        my $decode = $param->{decode} // 1;
 
         my $vmid = $param->{vmid};
         my $conf = PVE::QemuConfig->load_config($vmid);
@@ -515,7 +525,10 @@ __PACKAGE__->register_method({
                 mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($chunk_size));
             check_agent_error($read, "can't read from file");
 
-            $content .= decode_base64($read->{'buf-b64'});
+            my $chunk = $read->{'buf-b64'};
+            $chunk = decode_base64($chunk) if $decode;
+            $content .= $chunk;
+
             $bytes_read += $read->{count};
             $eof = $read->{eof} // 0;
         }
-- 
2.53.0

[PATCH qemu-server v2 3/3] agent: file-read: Allow specifying byte offset to start reading at

[Markus Ebner]

The previous implementation only allowed reads of fixed size starting
at byte offset 0. This made the Proxmox's agent/file-read unsuitable
for reading large files.
The new offset parameter allows specifying a byte offset position
relative to the start of the file (offset 0) from which the following
read should be performed. The implementation's behavior follows the
lseek semantics - directly following qemu-guest-agent's behavior:

- fseek() does not perform validation against the file size
- Seeking beyond the end of the file is not an error but explicitly
  allowed and specified behavior
- Subsequent fread() calls at positions beyond the file size return
  zero bytes

Additionally, this is equivalent to the behavior of POSIX pread().
Reading a large file block-by-block thus becomes (pseudo-code):
let content = [];
while(response.truncated) {
    response = proxmox.agent_file_read(offset = content.length);
    content .= response.content;
}

To be backwards compatible, offset defaults to 0.

Signed-off-by: Markus Ebner <info@ebner-markus.de>
---
 src/PVE/API2/Qemu/Agent.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm
index 1f5962e9..ba96d77e 100644
--- a/src/PVE/API2/Qemu/Agent.pm
+++ b/src/PVE/API2/Qemu/Agent.pm
@@ -485,6 +485,13 @@ __PACKAGE__->register_method({
                 type => 'string',
                 description => 'The path to the file',
             },
+            offset => {
+                type => 'integer',
+                optional => 1,
+                minimum => 0,
+                default => 0,
+                description => "Offset to start reading at",
+            },
         },
     },
     returns => {
@@ -506,6 +513,7 @@ __PACKAGE__->register_method({
         my ($param) = @_;
         my $count = int($param->{count} // $MAX_READ_SIZE);
         my $decode = $param->{decode} // 1;
+        my $offset = $param->{offset} // 0;
 
         my $vmid = $param->{vmid};
         my $conf = PVE::QemuConfig->load_config($vmid);
@@ -513,6 +521,16 @@ __PACKAGE__->register_method({
         my $qgafh =
             agent_cmd($vmid, $conf, "file-open", { path => $param->{file} }, "can't open file");
 
+        if ($offset > 0) {
+            my $seek = mon_cmd(
+                $vmid, "guest-file-seek",
+                handle => $qgafh,
+                offset => int($offset),
+                whence => 'set',
+            );
+            check_agent_error($seek, "can't seek to offset position");
+        }
+
         my $bytes_read = 0;
         my $eof = 0;
         my $read_size = 1024 * 1024;
-- 
2.53.0

Re: [PATCH qemu-server v2 3/3] agent: file-read: Allow specifying byte offset to start reading at

[Fiona Ebner]

Am 25.02.26 um 1:06 PM schrieb Markus Ebner:
> The previous implementation only allowed reads of fixed size starting
> at byte offset 0. This made the Proxmox's agent/file-read unsuitable
> for reading large files.
> The new offset parameter allows specifying a byte offset position
> relative to the start of the file (offset 0) from which the following
> read should be performed. The implementation's behavior follows the
> lseek semantics - directly following qemu-guest-agent's behavior:
> 
> - fseek() does not perform validation against the file size
> - Seeking beyond the end of the file is not an error but explicitly
>   allowed and specified behavior
> - Subsequent fread() calls at positions beyond the file size return
>   zero bytes
> 
> Additionally, this is equivalent to the behavior of POSIX pread().
> Reading a large file block-by-block thus becomes (pseudo-code):
> let content = [];
> while(response.truncated) {
>     response = proxmox.agent_file_read(offset = content.length);
>     content .= response.content;
> }
> 
> To be backwards compatible, offset defaults to 0.
> 
> Signed-off-by: Markus Ebner <info@ebner-markus.de>

Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>

Re: [PATCH qemu-server v2 2/3] agent: file-read: Allow maintaining base64-encoding of content

[Fiona Ebner]

Am 25.02.26 um 1:06 PM schrieb Markus Ebner:
> The previous implementation always decoded the base64-encoded content
> received from the qemu-guest-agent file-read call. Since JSON strings
> must be comliant unicode text, binary data got escaped using unicode
> escape sequences, using the pattern: \u00XX - with XX being the hex
> value of the byte to encode. For certain binary files, this lead to a
> massively inflated payload size of the API response.
> 
> Comparison on my test system:
> For a 4MiB test-file generated using
> dd if=/dev/urandom bs=4M count=1
> 
> - Reading it with decode=1 transfers 8.61MiB and takes 5700ms on avg.
> - Reading it with decode=0 transfers 5.59MiB and takes 3300ms on avg.
> 
> To be backwards compatible, the decode parameter defaults to 1.
> 
> Signed-off-by: Markus Ebner <info@ebner-markus.de>

Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>

Re: [PATCH qemu-server v2 1/3] agent: file-read: Allow specifying max number of bytes to read

[Fiona Ebner]

Am 25.02.26 um 1:06 PM schrieb Markus Ebner:
> The previous file-read implementation in the Proxmox API always
> attempted to read a fixed 16 MiB. On busy or resource-constrainted
> hosts, that often caused request timeouts, with no option to
> reduce the number of read bytes to at least read something.
> 
> The new parameter count now allows specifying the exact number of
> bytes that should be read from the given file. To be backwards
> compatible with the previous behavior, it defaults to 16 MiB.
> These 16 MiB are further also the maximum allowed value.
> 
> Signed-off-by: Markus Ebner <info@ebner-markus.de>
> ---
>  src/PVE/API2/Qemu/Agent.pm | 27 ++++++++++++++++++++-------
>  1 file changed, 20 insertions(+), 7 deletions(-)
> 
> diff --git a/src/PVE/API2/Qemu/Agent.pm b/src/PVE/API2/Qemu/Agent.pm
> index de36ce1e..f32e1dc2 100644
> --- a/src/PVE/API2/Qemu/Agent.pm
> +++ b/src/PVE/API2/Qemu/Agent.pm
> @@ -464,6 +464,14 @@ __PACKAGE__->register_method({
>                  'pve-vmid',
>                  { completion => \&PVE::QemuServer::complete_vmid_running },
>              ),
> +            count => {
> +                type => 'integer',
> +                optional => 1,
> +                minimum => 0,

Is allowing 0 intentional?

> +                maximum => $MAX_READ_SIZE,
> +                default => $MAX_READ_SIZE,
> +                description => "Number of bytes to read.",
> +            },
>              file => {
>                  type => 'string',
>                  description => 'The path to the file',
> @@ -487,6 +495,7 @@ __PACKAGE__->register_method({
>      },
>      code => sub {
>          my ($param) = @_;
> +        my $count = int($param->{count} // $MAX_READ_SIZE);

Nit: The int() is superfluous.

>  
>          my $vmid = $param->{vmid};
>          my $conf = PVE::QemuConfig->load_config($vmid);
> @@ -494,18 +503,20 @@ __PACKAGE__->register_method({
>          my $qgafh =
>              agent_cmd($vmid, $conf, "file-open", { path => $param->{file} }, "can't open file");
>  
> -        my $bytes_left = $MAX_READ_SIZE;
> +        my $bytes_read = 0;
>          my $eof = 0;
>          my $read_size = 1024 * 1024;
>          my $content = "";
>  
> -        while ($bytes_left > 0 && !$eof) {
> +        while ($bytes_read < $count && !$eof) {
> +            my $bytes_left = $count - $bytes_read;
> +            my $chunk_size = $bytes_left < $read_size ? $bytes_left : $read_size;
>              my $read =
> -                mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($read_size));
> +                mon_cmd($vmid, "guest-file-read", handle => $qgafh, count => int($chunk_size));
>              check_agent_error($read, "can't read from file");
>  
>              $content .= decode_base64($read->{'buf-b64'});
> -            $bytes_left -= $read->{count};
> +            $bytes_read += $read->{count};
>              $eof = $read->{eof} // 0;
>          }
>  
> @@ -514,12 +525,14 @@ __PACKAGE__->register_method({
>  
>          my $result = {
>              content => $content,
> -            'bytes-read' => ($MAX_READ_SIZE - $bytes_left),
> +            'bytes-read' => $bytes_read,
>          };
>  
>          if (!$eof) {
> -            warn
> -                "agent file-read: reached maximum read size: $MAX_READ_SIZE bytes. output might be truncated.\n";
> +            if (!defined $param->{count}) {

Style nit: please use parentheses for definedness checks:
if (!defined($param->{count})) {

> +                warn "agent file-read: reached maximum read size: $MAX_READ_SIZE bytes."
> +                    . " output might be truncated.\n";
> +            }
>              $result->{truncated} = 1;

The description for 'truncated' is:
"If set to 1, the output is truncated and not complete"

But if I request 1 byte and get 1 byte, why should it be considered
"truncated and not complete"?

I do think it's useful information to have 'truncated' for the API
users, but the description should be updated to better convey that it's
about whether EOF was reached or not.