[pve-devel] [PATCH proxmox-i18n] js generator: try to extract comments from source

[pve-devel] [PATCH proxmox-i18n] js generator: try to extract comments from source

[Maximiliano Sandoval]

Adds a way to extract comments to the resulting .po files matching
xgettext's behaviour. Useful for instances like

```js
   // Translators: This is the verb, not the noun
   gettext("Profile");

   // Translators: This would read 'Manage OSD'
   Ext.String.format(gettext('Manage {0}'), 'OSD');
```

where the string is not enough to guarantee is a satisfactory
translation.

Do note that two identical messages with different comments will count
as the same message (same msgid) from the point of view of gettext. To
truly differentiate them one would need to support Context, see [1].

Caveats:
  - Cannot extract multiline comments
  - Does not understand comments in the /* comment */ form

[1] https://www.gnu.org/software/gettext/manual/html_node/Contexts.html

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
---
 jsgettext.pl | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/jsgettext.pl b/jsgettext.pl
index 7f758fd..d0bf7a9 100755
--- a/jsgettext.pl
+++ b/jsgettext.pl
@@ -93,7 +93,7 @@ my $href = {
 };
 
 sub extract_msg {
-    my ($filename, $linenr, $line) = @_;
+    my ($filename, $linenr, $line, $comment) = @_;
 
     my $count = 0;
 
@@ -111,7 +111,7 @@ sub extract_msg {
 	if (my $po = $href->{$text}) {
 	    $po->reference($po->reference() . " $ref");
 	} else {
-	    $href->{$text} = Locale::PO->new(-msgid=> $text, -reference=> $ref, -msgstr=> '');
+	    $href->{$text} = Locale::PO->new(-msgid=> $text, -reference=> $ref, -msgstr=> '', -automatic=> $comment);
 	}
     }
     die "can't extract gettext message in '$filename' line $linenr\n" if !$count;
@@ -122,10 +122,13 @@ my $sources = find_js_sources($dirs);
 
 foreach my $s (@$sources) {
     open(my $SRC_FH, '<', $s) || die "unable to open file '$s' - $!\n";
+    my $prev_line;
     while(defined(my $line = <$SRC_FH>)) {
 	if ($line =~ m/gettext\s*\(/ && $line !~ m/^\s*function gettext/) {
-	    extract_msg($s, $., $line);
+	    my ($comment) = $prev_line =~ /\/\/\s+(.*)/;
+	    extract_msg($s, $., $line, $comment);
 	}
+	$prev_line = $line;
     }
     close($SRC_FH);
 }
-- 
2.39.2

Re: [pve-devel] [PATCH proxmox-i18n] js generator: try to extract comments from source

[Thomas Lamprecht]

Am 30/08/2023 um 14:48 schrieb Maximiliano Sandoval:
> Adds a way to extract comments to the resulting .po files matching
> xgettext's behaviour. Useful for instances like
> 
> ```js
>    // Translators: This is the verb, not the noun

hmm, but is then grouped with the reference comment in the resulting
PO file?

And even if, what use is the comment if there are then a usage as noun
and one as verb? For that we really would need the msgcontext stuff and
a pgettext implementation so that one could then actually do different
translations for either variant (e.g., in German then Noun -> "Profil"
and verb -> "Profilieren")

IMO the comments mostly make sense for describing string formats, but
support for them can be still fine as it doesn't cost much.

Oh, and while at it, I pondered over adding a wrapper for the common
Ext.String.format(gettext(MSG), PARAMS)) combi, something like
gettextf(MSG, PARAMS) or even shorter i18n(MSG, PARAMS) (with params
optional). This way we might even parse the passed params and add those
as comment automatically.

>    gettext("Profile");
> 
>    // Translators: This would read 'Manage OSD'
>    Ext.String.format(gettext('Manage {0}'), 'OSD');
> ```
> 
> where the string is not enough to guarantee is a satisfactory
> translation.
> 
> Do note that two identical messages with different comments will count
> as the same message (same msgid) from the point of view of gettext. To
> truly differentiate them one would need to support Context, see [1].
> 
> Caveats:
>   - Cannot extract multiline comments
>   - Does not understand comments in the /* comment */ form

those two are fine, but comments at the end of line won't be recognized either,
e.g.:

    fieldLabel: gettext('Foo'), // Translators: bar baz

No biggie, but IMO not to hard to parse, especially as we don't have
to bother that much for performance as this is only triggered manually
when updating the POT catalogue.

> 
> [1] https://www.gnu.org/software/gettext/manual/html_node/Contexts.html
> 
> Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
> ---
>  jsgettext.pl | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/jsgettext.pl b/jsgettext.pl
> index 7f758fd..d0bf7a9 100755
> --- a/jsgettext.pl
> +++ b/jsgettext.pl
> @@ -93,7 +93,7 @@ my $href = {
>  };
>  
>  sub extract_msg {
> -    my ($filename, $linenr, $line) = @_;
> +    my ($filename, $linenr, $line, $comment) = @_;
>  
>      my $count = 0;
>  
> @@ -111,7 +111,7 @@ sub extract_msg {
>  	if (my $po = $href->{$text}) {
>  	    $po->reference($po->reference() . " $ref");
>  	} else {
> -	    $href->{$text} = Locale::PO->new(-msgid=> $text, -reference=> $ref, -msgstr=> '');
> +	    $href->{$text} = Locale::PO->new(-msgid=> $text, -reference=> $ref, -msgstr=> '', -automatic=> $comment);
>  	}
>      }
>      die "can't extract gettext message in '$filename' line $linenr\n" if !$count;
> @@ -122,10 +122,13 @@ my $sources = find_js_sources($dirs);
>  
>  foreach my $s (@$sources) {
>      open(my $SRC_FH, '<', $s) || die "unable to open file '$s' - $!\n";
> +    my $prev_line;
>      while(defined(my $line = <$SRC_FH>)) {
>  	if ($line =~ m/gettext\s*\(/ && $line !~ m/^\s*function gettext/) {
> -	    extract_msg($s, $., $line);
> +	    my ($comment) = $prev_line =~ /\/\/\s+(.*)/;
> +	    extract_msg($s, $., $line, $comment);
>  	}
> +	$prev_line = $line;
>      }
>      close($SRC_FH);
>  }

Re: [pve-devel] [PATCH proxmox-i18n] js generator: try to extract comments from source

[Maximiliano Sandoval]

I did a quick test with

```js
     // Comment 1
     a: gettext('TEST 1'),
     // Comment 2
     b: gettext('TEST 1'),

     c: gettext('TEST 2'),
     // Comment 3
     d: gettext('TEST 2'),

     // Comment 4
     e: gettext('TEST 3'),
     f: gettext('TEST 3'),
```

and the resulting file contains a single instance of each msgid:

```

#. Comment 1
#: pve-manager/www/manager6/ceph/OSD.js:239
#: pve-manager/www/manager6/ceph/OSD.js:241
msgid "TEST 1"
msgstr ""

#: pve-manager/www/manager6/ceph/OSD.js:243
#: pve-manager/www/manager6/ceph/OSD.js:245
msgid "TEST 2"
msgstr ""

#. Comment 4
#: pve-manager/www/manager6/ceph/OSD.js:248
#: pve-manager/www/manager6/ceph/OSD.js:249
msgid "TEST 3"
msgstr ""
```

Compare with the output from xgettext (xgettext --language=JavaScript --add-comments OSD.js)
```
#. Comment 1
#. Comment 2
#: OSD.js:239 OSD.js:241
msgid "TEST 1"
msgstr ""

#. Comment 3
#: OSD.js:243 OSD.js:245
msgid "TEST 2"
msgstr ""

#. Comment 4
#: OSD.js:248 OSD.js:249
msgid "TEST 3"
msgstr ""
```

which means that at worst we will lose a comment depending on which msgid was
found first. Given that this is not using the pgettext method that takes into
account the to generate different translations, this seems fine to me, at least
as a first step into getting feature parity with (x)gettext.

On 8/30/23 15:23, Thomas Lamprecht wrote:
> Am 30/08/2023 um 14:48 schrieb Maximiliano Sandoval:
>> Adds a way to extract comments to the resulting .po files matching
>> xgettext's behaviour. Useful for instances like
>>
>> ```js
>>     // Translators: This is the verb, not the noun
> hmm, but is then grouped with the reference comment in the resulting
> PO file?
>
> And even if, what use is the comment if there are then a usage as noun
> and one as verb? For that we really would need the msgcontext stuff and
> a pgettext implementation so that one could then actually do different
> translations for either variant (e.g., in German then Noun -> "Profil"
> and verb -> "Profilieren")
>
> IMO the comments mostly make sense for describing string formats, but
> support for them can be still fine as it doesn't cost much.
>
> Oh, and while at it, I pondered over adding a wrapper for the common
> Ext.String.format(gettext(MSG), PARAMS)) combi, something like
> gettextf(MSG, PARAMS) or even shorter i18n(MSG, PARAMS) (with params
> optional). This way we might even parse the passed params and add those
> as comment automatically.
>
>>     gettext("Profile");
>>
>>     // Translators: This would read 'Manage OSD'
>>     Ext.String.format(gettext('Manage {0}'), 'OSD');
>> ```
>>
>> where the string is not enough to guarantee is a satisfactory
>> translation.
>>
>> Do note that two identical messages with different comments will count
>> as the same message (same msgid) from the point of view of gettext. To
>> truly differentiate them one would need to support Context, see [1].
>>
>> Caveats:
>>    - Cannot extract multiline comments
>>    - Does not understand comments in the /* comment */ form
> those two are fine, but comments at the end of line won't be recognized either,
> e.g.:
>
>      fieldLabel: gettext('Foo'), // Translators: bar baz
>
> No biggie, but IMO not to hard to parse, especially as we don't have
> to bother that much for performance as this is only triggered manually
> when updating the POT catalogue.
>
>> [1] https://www.gnu.org/software/gettext/manual/html_node/Contexts.html
>>
>> Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
>> ---
>>   jsgettext.pl | 9 ++++++---
>>   1 file changed, 6 insertions(+), 3 deletions(-)
>>
>> diff --git a/jsgettext.pl b/jsgettext.pl
>> index 7f758fd..d0bf7a9 100755
>> --- a/jsgettext.pl
>> +++ b/jsgettext.pl
>> @@ -93,7 +93,7 @@ my $href = {
>>   };
>>   
>>   sub extract_msg {
>> -    my ($filename, $linenr, $line) = @_;
>> +    my ($filename, $linenr, $line, $comment) = @_;
>>   
>>       my $count = 0;
>>   
>> @@ -111,7 +111,7 @@ sub extract_msg {
>>   	if (my $po = $href->{$text}) {
>>   	    $po->reference($po->reference() . " $ref");
>>   	} else {
>> -	    $href->{$text} = Locale::PO->new(-msgid=> $text, -reference=> $ref, -msgstr=> '');
>> +	    $href->{$text} = Locale::PO->new(-msgid=> $text, -reference=> $ref, -msgstr=> '', -automatic=> $comment);
>>   	}
>>       }
>>       die "can't extract gettext message in '$filename' line $linenr\n" if !$count;
>> @@ -122,10 +122,13 @@ my $sources = find_js_sources($dirs);
>>   
>>   foreach my $s (@$sources) {
>>       open(my $SRC_FH, '<', $s) || die "unable to open file '$s' - $!\n";
>> +    my $prev_line;
>>       while(defined(my $line = <$SRC_FH>)) {
>>   	if ($line =~ m/gettext\s*\(/ && $line !~ m/^\s*function gettext/) {
>> -	    extract_msg($s, $., $line);
>> +	    my ($comment) = $prev_line =~ /\/\/\s+(.*)/;
>> +	    extract_msg($s, $., $line, $comment);
>>   	}
>> +	$prev_line = $line;
>>       }
>>       close($SRC_FH);
>>   }

Re: [pve-devel] [PATCH proxmox-i18n] js generator: try to extract comments from source

[Maximiliano Sandoval]

On 8/30/23 15:23, Thomas Lamprecht wrote:
> Oh, and while at it, I pondered over adding a wrapper for the common Ext.String.format(gettext(MSG), PARAMS)) combi, something like gettextf(MSG, PARAMS) or even shorter i18n(MSG, PARAMS) (with params optional). This way we might even parse the passed params and add those as comment automatically.


Regarding this, sounds like a good idea for a future patch.