[pve-devel] [PATCH installer] country.pl: load iso 3166 country codes file as UTF8

[pve-devel] [PATCH installer] country.pl: load iso 3166 country codes file as UTF8

[Christoph Heiss]

This file is UTF-8 encoded and contains Unicode characters for encoding
country names, such as "Åland Islands" for example.

Otherwise, the final locale-info.json that gets shipped with the ISO
might contain wrongly-encoded country names, causing display errors in
(at least) the GTK installer country dropdown.

Reported-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
---
 country.pl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/country.pl b/country.pl
index 9e4881a..7401f68 100755
--- a/country.pl
+++ b/country.pl
@@ -88,7 +88,7 @@ my sub parse_zoneinfo {
 # country codes from:
 my $country_codes_file = "/usr/share/iso-codes/json/iso_3166-1.json";
 
-my $iso_3166_codes = from_json(PVE::Tools::file_get_contents($country_codes_file, 64 * 1024));
+my $iso_3166_codes = from_json(PVE::Tools::file_get_contents($country_codes_file, 64 * 1024), { utf8 => 1 });
 
 my $country_codes = { map { lc($_->{'alpha_2'}) => $_->{'common_name'} // $_->{'name'} } @{$iso_3166_codes->{'3166-1'}} };
 
-- 
2.48.1



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH installer] country.pl: load iso 3166 country codes file as UTF8

[Maximiliano Sandoval]

I ran
```sh
make locale-info.json
<locale-info.json jq > locales.json
```
then did a diff of the locales.json file before and after this patch. The resulting diff is:

```diff
@@ -1001,7 +1001,7 @@
     "ax": {
       "kmap": "",
       "mirror": "",
-      "name": "Ã…land Islands",
+      "name": "Åland Islands",
       "zone": "Europe/Mariehamn"
     },
     "az": {
@@ -1067,7 +1067,7 @@
     "bl": {
       "kmap": "",
       "mirror": "",
-      "name": "Saint Barthélemy",
+      "name": "Saint Barthélemy",
       "zone": "America/St_Barthelemy"
     },
     "bm": {
@@ -1174,7 +1174,7 @@
     "ci": {
       "kmap": "",
       "mirror": "",
-      "name": "Côte d'Ivoire",
+      "name": "Côte d'Ivoire",
       "zone": "Africa/Abidjan"
     },
     "ck": {
@@ -1228,7 +1228,7 @@
     "cw": {
       "kmap": "",
       "mirror": "",
-      "name": "Curaçao",
+      "name": "Curaçao",
       "zone": "America/Curacao"
     },
     "cx": {
@@ -2037,7 +2037,7 @@
     "re": {
       "kmap": "",
       "mirror": "",
-      "name": "Réunion",
+      "name": "Réunion",
       "zone": "Indian/Reunion"
     },
     "ro": {
@@ -2259,7 +2259,7 @@
     "tr": {
       "kmap": "",
       "mirror": "ftp.tr.debian.org",
-      "name": "Türkiye",
+      "name": "Türkiye",
       "zone": "Europe/Istanbul"
     },
     "tt": {
@@ -2463,10 +2463,10 @@
     "costa rica": "cr",
     "croatia": "hr",
     "cuba": "cu",
-    "curaã§ao": "cw",
+    "curaçao": "cw",
     "cyprus": "cy",
     "czechia": "cz",
-    "cã´te d'ivoire": "ci",
+    "côte d'ivoire": "ci",
     "denmark": "dk",
     "djibouti": "dj",
     "dominica": "dm",
@@ -2590,8 +2590,8 @@
     "romania": "ro",
     "russian federation": "ru",
     "rwanda": "rw",
-    "rã©union": "re",
-    "saint barthã©lemy": "bl",
+    "réunion": "re",
+    "saint barthélemy": "bl",
     "saint helena, ascension and tristan da cunha": "sh",
     "saint kitts and nevis": "kn",
     "saint lucia": "lc",
@@ -2637,7 +2637,7 @@
     "turkmenistan": "tm",
     "turks and caicos islands": "tc",
     "tuvalu": "tv",
-    "tã¼rkiye": "tr",
+    "türkiye": "tr",
     "uganda": "ug",
     "ukraine": "ua",
     "united arab emirates": "ae",
@@ -2656,7 +2656,7 @@
     "yemen": "ye",
     "zambia": "zm",
     "zimbabwe": "zw",
-    "ã…land islands": "ax"
+    "åland islands": "ax"
   },
   "kmap": {
     "de": {
```

which seems correct to me.

Thanks for the patch!

Reviewed-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>

Christoph Heiss <c.heiss@proxmox.com> writes:

> This file is UTF-8 encoded and contains Unicode characters for encoding
> country names, such as "Åland Islands" for example.
>
> Otherwise, the final locale-info.json that gets shipped with the ISO
> might contain wrongly-encoded country names, causing display errors in
> (at least) the GTK installer country dropdown.
>
> Reported-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
> Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
> ---
>  country.pl | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/country.pl b/country.pl
> index 9e4881a..7401f68 100755
> --- a/country.pl
> +++ b/country.pl
> @@ -88,7 +88,7 @@ my sub parse_zoneinfo {
>  # country codes from:
>  my $country_codes_file = "/usr/share/iso-codes/json/iso_3166-1.json";
>  
> -my $iso_3166_codes = from_json(PVE::Tools::file_get_contents($country_codes_file, 64 * 1024));
> +my $iso_3166_codes = from_json(PVE::Tools::file_get_contents($country_codes_file, 64 * 1024), { utf8 => 1 });
>  
>  my $country_codes = { map { lc($_->{'alpha_2'}) => $_->{'common_name'} // $_->{'name'} } @{$iso_3166_codes->{'3166-1'}} };



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] Applied: [PATCH installer] country.pl: load iso 3166 country codes file as UTF8

[Thomas Lamprecht]

On Tue, 08 Apr 2025 10:47:43 +0200, Christoph Heiss wrote:
> This file is UTF-8 encoded and contains Unicode characters for encoding
> country names, such as "Åland Islands" for example.
> 
> Otherwise, the final locale-info.json that gets shipped with the ISO
> might contain wrongly-encoded country names, causing display errors in
> (at least) the GTK installer country dropdown.
> 
> [...]

Exactly what I thought it would be when Maximiliano showed me the screenshot of
the broken encoding, hindsigth is 20/20 I guess.

Applied, thanks!

[1/1] country.pl: load iso 3166 country codes file as UTF8
      commit: 4b5db269ec8aae689329633a1ebe9546a19490b3


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel