From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E1AC39DD41 for ; Tue, 6 Jun 2023 11:12:52 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BADAD32210 for ; Tue, 6 Jun 2023 11:12:22 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 6 Jun 2023 11:12:22 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id AB35F48C67 for ; Tue, 6 Jun 2023 11:12:21 +0200 (CEST) Message-ID: <7f0da808-115b-6f31-2cf2-3bd3f0e7e27b@proxmox.com> Date: Tue, 6 Jun 2023 11:12:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-GB, de-AT To: Proxmox VE development discussion , Dominik Csapak References: <20230606083914.1400960-1-d.csapak@proxmox.com> <20230606083914.1400960-2-d.csapak@proxmox.com> From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <20230606083914.1400960-2-d.csapak@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.085 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pve-devel] [PATCH common v2 1/3] JSONSchema: add support for array parameter in api calls, cli and config X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2023 09:12:52 -0000 Am 06/06/2023 um 10:39 schrieb Dominik Csapak: > a few things were missing for it to work: > * on the cli, we have to get the option as an array if the type is an > array > * the untainting must be done recursively, otherwise, the regex matching > converts an array hash into the string 'ARRAY(0x123412341234)' > * JSONSchema::parse_config did not handle array formats specially, but > we want to allow to specify them multiple time > * the biggest point: in the RESTHandler, to be compatible with the > current gui behavior, we have to rewrite two parameter types: > - when the api defines a '-list' format for a string type, but we get > a list (because of the changes in http-server), we join the list > with a comma into a string > - when the api defines an 'array' type, but we get a scalar value, > wrap the value in an array (because for www-form-urlencoded, you > cannot send an array with a single value) add tests for this > behavior, some of which we want to deprecate and remove in the > future > > Signed-off-by: Dominik Csapak > --- > changes from v1: > * include wolfangs feedback > * include auto-conversion from string <-> list where appropriate and add > tests for it > > src/PVE/JSONSchema.pm | 12 +++++ > src/PVE/RESTHandler.pm | 61 ++++++++++++++++++---- > test/Makefile | 9 +++- > test/api_parameter_test.pl | 100 +++++++++++++++++++++++++++++++++++++ > 4 files changed, 172 insertions(+), 10 deletions(-) > create mode 100755 test/api_parameter_test.pl > > diff --git a/src/PVE/JSONSchema.pm b/src/PVE/JSONSchema.pm > index 527e409..526fc2b 100644 > --- a/src/PVE/JSONSchema.pm > +++ b/src/PVE/JSONSchema.pm > @@ -1709,6 +1709,8 @@ sub get_options { > } else { > if ($pd->{format} && $pd->{format} =~ m/-a?list/) { > push @getopt, "$prop=s@"; > + } elsif ($pd->{type} eq 'array') { > + push @getopt, "$prop=s@"; > } else { > push @getopt, "$prop=s"; > } > @@ -1869,6 +1871,16 @@ sub parse_config : prototype($$$;$) { > > $value = parse_boolean($value) // $value; > } > + if ($schema->{properties}->{$key} && > + $schema->{properties}->{$key}->{type} eq 'array') { code style, and can be fixed up: for multi-line if's place the closing parenthesis and opening block { on it's own line: It also doesn't hurt to move all expressions part of the condition in a separate line (albeit that part is not a rule in our style guide): if ( $schema->{properties}->{$key} && $schema->{properties}->{$key}->{type} eq 'array' ) { # ... > + > + if (defined($cfg->{$key})) { > + push $cfg->{$key}->@*, $value; > + } else { > + $cfg->{$key} = [$value]; > + } Could be written shorter, but just fine as above $cfg->{$key} //= []; push $cfg->{$key}->@*, $value; > + next; > + } > $cfg->{$key} = $value; > } else { > warn "ignore config line: $line\n" > diff --git a/src/PVE/RESTHandler.pm b/src/PVE/RESTHandler.pm > index db86af2..369e302 100644 > --- a/src/PVE/RESTHandler.pm > +++ b/src/PVE/RESTHandler.pm > @@ -426,6 +426,56 @@ sub find_handler { > return ($handler_class, $method_info); > } > > +my $untaint_recursive; I got flash backs w.r.t. refcount cycles here keeping all variables, and thus memory inside the body alive forever, don't we need a weaken? E.g., like we had to do in PVE::Status::Graphite's assemble. > + > +$untaint_recursive = sub { > + my ($param) = @_; > + > + my $ref = ref($param); > + if ($ref eq 'HASH') { > + $param->{$_} = $untaint_recursive->($param->{$_}) for keys $param->%*; > + } elsif ($ref eq 'ARRAY') { > + for (my $i = 0; $i < scalar($param->@*); $i++) { > + $param->[$i] = $untaint_recursive->($param->[$i]); > + } > + } else { > + if (defined($param)) { could be merged into upper branch as elsif, but no hard feelings. > + my ($newval) = $param =~ /^(.*)$/s; > + $param = $newval; > + } > + } > + > + return $param; > +}; > + > +# convert arrays to strings where we expect a '-list' format and convert scalar > +# values to arrays when we expect an array (because of www-form-urlencoded) > +# > +# only on the top level, since www-form-urlencoded cannot be nested anyway > +# > +# FIXME: change gui/api calls to not rely on this during 8.x, mark the > +# behaviour deprecated with 9.x, and remove it with 10.x > +my $convert_params = sub { my ($param, $schema) = @_; please keep the method paramethers on it's own line. Also, maybe go for a more telling names, as convert_params could mean everytrhing and nothing ^^ > + > + return if !$schema->{properties}; > + return if (ref($param) // '') ne 'HASH'; doesn't this breaks the assignment when used below? I.e.,: $param = $convert_params->($param, $schema); or messes with silenting parameters sent to a endpoint without properties, which would create an extra param error otherwise? > + > + for my $key (keys $schema->{properties}->%*) { > + if (my $value = $param->{$key}) { > + my $type = $schema->{properties}->{$key}->{type} // ''; > + my $format = $schema->{properties}->{$key}->{format} // ''; > + my $ref = ref($value); > + if ($ref eq 'ARRAY' && $type eq 'string' && $format =~ m/-list$/) { Should this also check ref to not be undef, i.e. if ($ref && $ref eq 'ARRAY' && ... > + $param->{$key} = join(',', $value->@*); > + } elsif (!$ref && $type eq 'array') { > + $param->{$key} = [$value]; > + } > + } > + } > + > + return $param; > +}; > + > sub handle { > my ($self, $info, $param, $result_verification) = @_; > > @@ -437,17 +487,10 @@ sub handle { > > if (my $schema = $info->{parameters}) { > # warn "validate ". Dumper($param}) . "\n" . Dumper($schema); > + $param = $convert_params->($param, $schema); > PVE::JSONSchema::validate($param, $schema); > # untaint data (already validated) > - my $extra = delete $param->{'extra-args'}; > - while (my ($key, $val) = each %$param) { > - if (defined($val)) { > - ($param->{$key}) = $val =~ /^(.*)$/s; > - } else { > - $param->{$key} = undef; > - } > - } > - $param->{'extra-args'} = [map { /^(.*)$/ } @$extra] if $extra; > + $param = $untaint_recursive->($param); > } > > my $result = $func->($param); # the actual API code execution call