[PATCH common 0/2] cli formatter: fix double UTF-8 encoding of YAML

[PATCH common 0/2] cli formatter: fix double UTF-8 encoding of YAML

[Michal Fox]

both the 'yaml' output format and the 'yaml' renderer of the text output
still handled the output of YAML::XS::Dump as a character string, which
was right for CPAN::Meta::YAML used before 99d0252, so any non-ASCII
character got encoded twice.

1/2 fixes the 'yaml' output format from #6305, as the reporter suggested
in the bug. 2/2 does the same for the renderer, which is used for
example for the ACME account details.

tested with 'make check' in src and test, each patch on its own.

Michal Fox (2):
  fix #6305: cli formatter: avoid encoding YAML output twice
  cli formatter: decode the output of the yaml renderer

 src/PVE/CLIFormatter.pm |  6 ++++--
 test/format_test.pl     | 16 ++++++++++++++++
 2 files changed, 20 insertions(+), 2 deletions(-)

-- 
2.43.0

[PATCH common 1/2] fix #6305: cli formatter: avoid encoding YAML output twice

[Michal Fox]

YAML::XS::Dump already returns UTF-8 encoded octets, but its output
still got encoded once more. This is a leftover from when the YAML
output was generated by CPAN::Meta::YAML, which returns a character
string.

As a result, all non-ASCII characters got mangled with the 'yaml'
output format, for example a cluster description with a German umlaut
when running:

  pvesh get /cluster/options --output-format yaml

Print the output of YAML::XS::Dump as is, like it is already done for
the 'json' output format, and add a test for the encoding of both.

Fixes: 99d0252 ("properly encode YAML via YAML::XS")
Signed-off-by: Michal Fox <me@dualfroz.com>
---
 src/PVE/CLIFormatter.pm |  3 ++-
 test/format_test.pl     | 15 +++++++++++++++
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/src/PVE/CLIFormatter.pm b/src/PVE/CLIFormatter.pm
index ca4ee08..d93bed6 100644
--- a/src/PVE/CLIFormatter.pm
+++ b/src/PVE/CLIFormatter.pm
@@ -380,7 +380,8 @@ sub print_api_result {
     }
 
     if ($format eq 'yaml') {
-        print encode('UTF-8', YAML::XS::Dump($data));
+        # Note: YAML::XS already returns UTF-8 encoded output
+        print YAML::XS::Dump($data);
     } elsif ($format eq 'json') {
         # Note: we always use utf8 encoding for json format
         print to_json($data, { utf8 => 1, allow_nonref => 1, canonical => 1 }) . "\n";
diff --git a/test/format_test.pl b/test/format_test.pl
index 7b461ef..8ccfd5a 100755
--- a/test/format_test.pl
+++ b/test/format_test.pl
@@ -7,6 +7,7 @@ use lib '../src';
 use PVE::JSONSchema;
 use PVE::CLIFormatter;
 
+use Encode qw(decode);
 use Test::More;
 use Test::MockModule;
 
@@ -53,4 +54,18 @@ foreach my $data (@$render_data) {
     is($actual, $expected, "string format '$renderer_name'");
 }
 
+# test that non-ASCII characters survive the structured output formats
+my $unicode_data = { description => "Erkl\x{e4}rung" };
+for my $format (qw(json yaml)) {
+    open(my $fh, '>', \my $output) or die "failed to open in-memory file - $!\n";
+    my $stdout = select($fh);
+    PVE::CLIFormatter::print_api_result(
+        $unicode_data, undef, undef, { 'output-format' => $format }, {},
+    );
+    select($stdout);
+    close($fh);
+
+    like(decode('UTF-8', $output), qr/Erkl\x{e4}rung/, "output format '$format' is UTF-8 encoded");
+}
+
 done_testing();
-- 
2.43.0

[PATCH common 2/2] cli formatter: decode the output of the yaml renderer

[Michal Fox]

YAML::XS::Dump returns UTF-8 encoded octets, but the value returned by
a renderer gets encoded when the text output is printed, like any other
value. So for properties using the 'yaml' renderer, for example the
account details shown by 'pvenode acme account info', any non-ASCII
character got encoded twice and ended up mangled.

Decode the output of YAML::XS::Dump in the renderer, so it returns a
character string like all other renderers, and add a test for it.

Fixes: 99d0252 ("properly encode YAML via YAML::XS")
Signed-off-by: Michal Fox <me@dualfroz.com>
---
 src/PVE/CLIFormatter.pm | 3 ++-
 test/format_test.pl     | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/PVE/CLIFormatter.pm b/src/PVE/CLIFormatter.pm
index d93bed6..758809e 100644
--- a/src/PVE/CLIFormatter.pm
+++ b/src/PVE/CLIFormatter.pm
@@ -27,7 +27,8 @@ PVE::JSONSchema::register_renderer('bytes', \&PVE::Format::render_bytes);
 sub render_yaml {
     my ($value) = @_;
 
-    my $data = YAML::XS::Dump($value);
+    # YAML::XS returns UTF-8 encoded output, but renderers need to return text
+    my $data = decode('UTF-8', YAML::XS::Dump($value));
     $data =~ s/^---[\n\s]//; # remove yaml marker
 
     return $data;
diff --git a/test/format_test.pl b/test/format_test.pl
index 8ccfd5a..2b9460d 100755
--- a/test/format_test.pl
+++ b/test/format_test.pl
@@ -45,6 +45,7 @@ my $render_data = [
     ["bytes", 1024, undef, "1.00 KiB"],
     ["bytes", 1024 * 1024 * 123 + 1024 * 300, 1, "123.3 MiB"],
     ["bytes", 1024 * 1024 * 1024 * 1024 * 4 + 1024 * 1024 * 2048 * 8, undef, "4.02 TiB"],
+    ["yaml", { description => "Erkl\x{e4}rung" }, undef, "description: Erkl\x{e4}rung\n"],
 ];
 
 foreach my $data (@$render_data) {
-- 
2.43.0