From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <m.carrara@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 8B33490769
for <pve-devel@lists.proxmox.com>; Tue, 2 Apr 2024 16:57:16 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 62428835A
for <pve-devel@lists.proxmox.com>; Tue, 2 Apr 2024 16:56:46 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[94.136.29.106])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Tue, 2 Apr 2024 16:56:45 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 181794476F
for <pve-devel@lists.proxmox.com>; Tue, 2 Apr 2024 16:56:45 +0200 (CEST)
From: Max Carrara <m.carrara@proxmox.com>
To: pve-devel@lists.proxmox.com
Date: Tue, 2 Apr 2024 16:55:20 +0200
Message-Id: <20240402145523.683008-9-m.carrara@proxmox.com>
X-Mailer: git-send-email 2.39.2
In-Reply-To: <20240402145523.683008-1-m.carrara@proxmox.com>
References: <20240402145523.683008-1-m.carrara@proxmox.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.022 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: [pve-devel] [PATCH v5 pve-storage 08/11] cephconfig: align our
parser with Ceph's parser
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 02 Apr 2024 14:57:16 -0000
This commit rewrites the entire parser for ceph.conf, aligning its
behaviour as closely as possible with Ceph's parser grammar [0].
The most notable improvements are as follows:
1. The characters '#' and ';' now both mark comments, instead of
just the '#' character.
2. Any character, including comment literals ('#' and ';'), may now
be escaped.
3. Quoted values (single and double) are now supported.
4. Line continuations are now supported (lines ending with '\').
5. Repeated whitespace characters in keys are now treated as a
single space character.
6. Dashes '-' are not treated the same as spaces and underscores
anymore, as Ceph's grammar doesn't treat them that way.
* Paired with 5., this means that repeated whitespace is now
equivalent to a single underscore.
7. Escaped comment literals are now un-escaped.
8. Although not too crucial, the parser now also supports empty
sections and will just initialize them with an empty hash.
Furthermore, the original grammar's more quirky behaviours are also
respected where sanely possible.
[0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=ceph/src/common/ConfUtils.cc;h=2f78fd02bf9e27467275752e6f3bca0c5e3946ce;hb=e9fe820e7fffd1b7cde143a9f77653b73fcec748#l144
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
---
Changes v1 --> v2:
* new
Changes v2 --> v3:
* support comment literals
Changes v3 --> v4:
* support empty sections
* fix and move support for comment literals to separate patch
Changes v4 --> v5:
* rewrite entire parsing logic in order to handle a lot more
(edge) cases
* support quoted values
* support line continuations wherever Ceph's grammar supports them
(in section headings, directly after the key, direcly after the
equals sign of kv-pairs, within unquoted values, directly after
quoted values, randomly within the file as long as the next line
consists of whitespace)
* support escaping any kind of character
* treat repeated whitespace as single space character (like Ceph does)
* use upfront defined regexes for readability
* use inner `sub`s for readability
* reword commit message
src/PVE/CephConfig.pm | 236 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 222 insertions(+), 14 deletions(-)
diff --git a/src/PVE/CephConfig.pm b/src/PVE/CephConfig.pm
index 0def1f2..56e7989 100644
--- a/src/PVE/CephConfig.pm
+++ b/src/PVE/CephConfig.pm
@@ -10,36 +10,244 @@ cfs_register_file('ceph.conf',
\&parse_ceph_config,
\&write_ceph_config);
+# For more information on how the Ceph parser works and how its grammar is
+# defined, see:
+# https://git.proxmox.com/?p=ceph.git;a=blob;f=ceph/src/common/ConfUtils.cc;h=2f78fd02bf9e27467275752e6f3bca0c5e3946ce;hb=e9fe820e7fffd1b7cde143a9f77653b73fcec748#l144
sub parse_ceph_config {
my ($filename, $raw) = @_;
my $cfg = {};
return $cfg if !defined($raw);
- my @lines = split /\n/, $raw;
+ # Note: According to Ceph's config grammar, a single key-value pair in a file
+ # (and nothing else!) is a valid config file and will be parsed by `ceph-conf`.
+ # We choose not to handle this case here because it doesn't seem to be used
+ # by Ceph at all (and otherwise doesn't really make sense anyway).
+
+ # Regexes ending with '_class' consist of only an extended character class
+ # each, which allows them to be interpolated into other ext. char classes.
+
+ my $re_leading_ws = qr/^\s+/;
+ my $re_trailing_ws = qr/\s+$/;
+
+ my $re_continue_marker = qr/\\/;
+ my $re_comment_class = qr/(?[ [ ; # ] ])/;
+ my $re_not_allowed_in_section_header_class = qr/(?[ [ \] ] + $re_comment_class ])/;
+
+ # Note: The Ceph config grammar defines keys the following way:
+ #
+ # key %= raw[+(text_char - char_("=[ ")) % +blank];
+ #
+ # The ' - char_("=[ ")' expression might lure you into thinking that keys
+ # may *not* contain spaces, but they can, due to the "% +blank" at the end!
+ #
+ # See: https://www.boost.org/doc/libs/1_42_0/libs/spirit/doc/html/spirit/qi/reference/operator/list.html
+ #
+ # Allowing spaces in this class and later squeezing whitespace as well as
+ # removing any leading and trailing whitespace from keys is just so much
+ # easier in our case.
+ my $re_not_allowed_in_keys_class = qr/(?[ [ = \[ ] + $re_comment_class ])/;
+
+ my $re_not_allowed_in_single_quoted_text_class = qr/(?[ [ ' ] + $re_comment_class ])/;
+ my $re_not_allowed_in_double_quoted_text_class = qr/(?[ [ " ] + $re_comment_class ])/;
+
+ my $re_text_char = qr/\\.|(?[ ! $re_comment_class ])/;
+ my $re_section_header_char = qr/\\.|(?[ ! $re_not_allowed_in_section_header_class ])/;
+
+ my $re_key_char = qr/\\.|(?[ ! $re_not_allowed_in_keys_class ])/;
+
+ my $re_single_quoted_text_char = qr/\\.|(?[ ! $re_not_allowed_in_single_quoted_text_class ])/;
+ my $re_double_quoted_text_char = qr/\\.|(?[ ! $re_not_allowed_in_double_quoted_text_class ])/;
+
+ my $re_single_quoted_value = qr/'(($re_single_quoted_text_char)*)'/;
+ my $re_double_quoted_value = qr/"(($re_double_quoted_text_char)*)"/;
+
+ my $re_key = qr/^(($re_key_char)+)/;
+ my $re_quoted_value = qr/$re_single_quoted_value|$re_double_quoted_value/;
+ my $re_unquoted_value = qr/(($re_text_char)*)/;
+ my $re_value = qr/($re_quoted_value|$re_unquoted_value)/;
+
+ my $re_kv_separator = qr/\s*(=)\s*/;
+
+ my $re_section_start = qr/\[/;
+ my $re_section_end = qr/\]/;
+ my $re_section_header = qr/$re_section_start(($re_section_header_char)+)$re_section_end/;
my $section;
+ my @lines = split(/\n/, $raw);
+
+ my $parse_section_header = sub {
+ my ($section_line) = @_;
+
+ # continued lines in section headers are allowed
+ while ($section_line =~ s/$re_continue_marker$//) {
+ $section_line .= shift(@lines);
+ }
+
+ my $remainder = $section_line;
+
+ $remainder =~ s/$re_section_header//;
+ my $parsed_header = $1;
+
+ # Un-escape comment literals
+ $parsed_header =~ s/\\($re_comment_class)/$1/g;
+
+ if (!$parsed_header) {
+ die "failed to parse section - skip: $section_line\n";
+ }
+
+ # preserve Ceph's behaviour and disallow anything after the section header
+ # that's not whitespace or a comment
+ $remainder =~ s/$re_leading_ws//;
+ $remainder =~ s/^$re_comment_class.*$//;
+
+ if ($remainder) {
+ die "unexpected remainder after section - skip: $section_line\n";
+ }
+
+ return $parsed_header;
+ };
+
+ my $parse_key = sub {
+ my ($line) = @_;
+
+ my $remainder = $line;
+
+ my $key = '';
+ while ($remainder =~ s/$re_key//) {
+ $key .= $1;
+
+ while ($key =~ s/$re_continue_marker$//) {
+ $remainder = shift(@lines);
+ }
+ }
+
+ $key =~ s/$re_trailing_ws//;
+ $key =~ s/$re_leading_ws//;
+
+ $key =~ s/\s/ /;
+ while ($key =~ s/\s\s/ /) {} # squeeze repeated whitespace
+
+ # Ceph treats *single* spaces in keys the same as underscores,
+ # but we'll just use underscores for readability
+ $key =~ s/ /_/g;
+
+ # Un-escape comment literals
+ $key =~ s/\\($re_comment_class)/$1/g;
+
+ if ($key eq '') {
+ die "failed to parse key from line - skip: $line\n";
+ }
+
+ my $had_equals = $remainder =~ s/^$re_kv_separator//;
+
+ if (!$had_equals) {
+ die "expected '=' after key - skip: $line\n";
+ }
+
+ while ($remainder =~ s/^$re_continue_marker$//) {
+ # Whitespace and continuations after equals sign can be arbitrary
+ $remainder = shift(@lines);
+ $remainder =~ s/$re_leading_ws//;
+ }
+
+ return ($key, $remainder);
+ };
+
+ my $parse_value = sub {
+ my ($line, $remainder) = @_;
+
+ my $starts_with_quote = $remainder =~ m/^['"]/;
+ $remainder =~ s/$re_value//;
+ my $value = $1 // '';
+
+ if ($value eq '') {
+ die "failed to parse value - skip: $line\n";
+ }
+
+ if ($starts_with_quote) {
+ # If it started with a quote, the parsed value MUST end with a quote
+ my $is_single_quoted = $value =~ m/$re_single_quoted_value/;
+ $value = $1 if $is_single_quoted;
+ my $is_double_quoted = !$is_single_quoted && $value =~ m/$re_double_quoted_value/;
+ $value = $1 if $is_double_quoted;
- foreach my $line (@lines) {
- $line =~ s/#.*$//;
- $line =~ s/^\s+//;
- $line =~ s/^;.*$//;
- $line =~ s/\s+$//;
+ if (!($is_single_quoted || $is_double_quoted)) {
+ die "failed to parse quoted value - skip: $line\n";
+ }
+
+ # Optionally, *only* line continuations may *only* follow right after
+ while ($remainder =~ s/^$re_continue_marker$//) {
+ $remainder .= shift(@lines);
+ }
+
+ # Nothing but whitespace or a comment may follow
+ $remainder =~ s/$re_leading_ws//;
+ $remainder =~ s/^$re_comment_class.*$//;
+
+ if ($remainder) {
+ die "unexpected remainder after value - skip: $line\n";
+ }
+
+ } else {
+ while ($value =~ s/$re_continue_marker$//) {
+ my $next_line = shift(@lines);
+
+ $next_line =~ s/$re_unquoted_value//;
+ my $value_part = $1 // '';
+ $value .= $value_part;
+ }
+
+ $value =~ s/$re_trailing_ws//;
+ }
+
+ # Un-escape comment literals
+ $value =~ s/\\($re_comment_class)/$1/g;
+
+ return $value;
+ };
+
+ while (scalar(@lines)) {
+ my $line = shift(@lines);
+
+ $line =~ s/^\s*(?<!\\)$re_comment_class.*$//;
+ $line =~ s/^\s*$//;
next if !$line;
+ next if $line =~ m/^$re_continue_marker$/;
+
+ if ($line =~ m/$re_section_start/) {
+ $section = undef;
+
+ eval { $section = $parse_section_header->($line) };
+ if ($@) {
+ warn "$@\n";
+ }
- $section = $1 if $line =~ m/^\[(\S+)\]$/;
- if (!$section) {
- warn "no section - skip: $line\n";
+ if (defined($section)) {
+ $cfg->{$section} = {} if !exists($cfg->{$section});
+ }
+
+ next;
+ }
+
+ if (!defined($section)) {
+ warn "no section header - skip: $line\n";
next;
}
- if ($line =~ m/^(.*?\S)\s*=\s*(\S.*)$/) {
- my ($key, $val) = ($1, $2);
- # ceph treats ' ', '_' and '-' in keys the same, so lets do too
- $key =~ s/[-\ ]/_/g;
- $cfg->{$section}->{$key} = $val;
+ my ($key, $remainder) = eval { $parse_key->($line) };
+ if ($@) {
+ warn "$@\n";
+ next;
+ }
+
+ my $value = eval { $parse_value->($line, $remainder) };
+ if ($@) {
+ warn "$@\n";
+ next;
}
+ $cfg->{$section}->{$key} = $value;
}
return $cfg;
--
2.39.2