From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C5272C0CA for ; Fri, 8 Apr 2022 20:25:57 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B83131A469 for ; Fri, 8 Apr 2022 20:25:57 +0200 (CEST) Received: from mx1.tetaneutral.net (mx1.tetaneutral.net [IPv6:2a03:7220:8081:5300::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 0FEA11A45E for ; Fri, 8 Apr 2022 20:25:56 +0200 (CEST) Received: by mx1.tetaneutral.net (Postfix, from userid 109) id 7379D8C0CB; Fri, 8 Apr 2022 20:25:55 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mx1.tetaneutral.net X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, FSL_HELO_NON_FQDN_1 autolearn=ham autolearn_force=no version=3.4.2 Received: from lgnuc (ip165.tetaneutral.net [IPv6:2a03:7220:8080:a500::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.tetaneutral.net (Postfix) with ESMTPS id F14CF8C023 for ; Fri, 8 Apr 2022 20:25:51 +0200 (CEST) Message-ID: <1649442351.6346.25.camel@guerby.net> From: Laurent GUERBY To: Proxmox VE development discussion Date: Fri, 08 Apr 2022 20:25:51 +0200 In-Reply-To: <1649408819.6346.22.camel@guerby.net> References: <1649408819.6346.22.camel@guerby.net> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.6-1+deb9u2 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pve-devel] New API endpoint to manage snippets / bugzilla 2208 / updated patch but missing something, need help X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2022 18:25:57 -0000 > Hi, > > I've tried to adapt the patch to current PVE 7.1-12 (see below) but I > still get > > proxmoxer.core.ResourceException: 501 Not Implemented: upload failed > - > {'errors': b''} > > When I try to upload a snippet. > > My proxmoxer setup works for iso, the following succeeds: > > proxmox.nodes(h).storage(s).upload.post(content="iso",filename=f) > > But the following fails (f being read "rb" from some xxx.yaml) > > proxmox.nodes(h).storage(s).upload.post(content="snippets",filename=f > ) > > Not having snippet upload makes it impossible to use PVE auth realm > tokens to control permissions, you have to give a priviledged system > account to users *just* to be able to do cloud init with a yaml which > is not great security wise (and not practical). > > I'm probably missing a few things to have a patch that works, I'm > willing to put some time on it if someone gives me directions. I instrumented some PVE perl code and failure to recognize the upload API data seems to happen somewhere in the logic of /usr/share/perl5/PVE/APIServer/AnyEvent.pm:file_upload_multipart With my patch installed for content=snippet filename=yoyobig.yaml if I upload a 16137 or more byte files it works, but for 16136 byte and below it fails with 501 not implemented which is suspicious. By adding max debug to python http libs I was able to check that the headers are strictly identical except for Content-Length as expected, see below. There's a comment in AnyEvent.pm about "# assume we have single line headers" and also some complex state handling while parsing the request but for now I wasn't able to pinpoint what triggers the failure. Help welcomed :) Sincerely, Laurent Debug logs: import http.client http.client.HTTPConnection.debuglevel=5 import logging logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) Works: INFO:proxmoxer.core:POST https://nuc2:8006/api2/json/nodes/nuc2/storage/element2dir/upload {'content': 'snippets', 'filename': <_io.BufferedReader name='yoyobig.yaml'>} DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): nuc2:8006 send: b'POST /api2/json/nodes/nuc2/storage/element2dir/upload HTTP/1.1\r\nHost: nuc2:8006\r\nUser-Agent: python-requests/2.23.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javasc ript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nContent-Length: 16385\r\nContent-Type: multipart/form-data; boundary=790f8b2d9dcdc452863bff66e1d262bc\r\nAuthorization: PVEAPITo ken=root@pam!testpython1=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\r\n\r\n' send: b'--790f8b2d9dcdc452863bff66e1d262bc\r\nContent-Disposition: form-data; name="content"\r\n\r\nsnippets\r\n--790f8b2d9dcdc452863bff66e1d262bc\r\nContent-Disposition: form-data; name="filename"; filename="yo yobig.yaml"\r\n\r\n#cloud-config\xhostname: test... Fails: INFO:proxmoxer.core:POST https://nuc2:8006/api2/json/nodes/nuc2/storage/element2dir/upload {'content': 'snippets', 'filename': <_io.BufferedReader name='yoyobig.yaml'>} DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): nuc2:8006 send: b'POST /api2/json/nodes/nuc2/storage/element2dir/upload HTTP/1.1\r\nHost: nuc2:8006\r\nUser-Agent: python-requests/2.23.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javasc ript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nContent-Length: 16384\r\nContent-Type: multipart/form-data; boundary=1ae9ce4d260e3f514d7c12ec03e4389f\r\nAuthorization: PVEAPITo ken=root@pam!testpython1=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\r\n\r\n' send: b'--1ae9ce4d260e3f514d7c12ec03e4389f\r\nContent-Disposition: form-data; name="content"\r\n\r\nsnippets\r\n--1ae9ce4d260e3f514d7c12ec03e4389f\r\nContent-Disposition: form-data; name="filename"; filename="yo yobig.yaml"\r\n\r\n#cloud-config\nhostname: test... > Thanks! > > Sincerely, > > Laurent (paying PVE+PBS customer at work) > > root@test:/usr/share/perl5# diff -u PVE/Storage.pm{-orig,} > --- PVE/Storage.pm-orig 2022-04-08 09:15:52.443943197 +0200 > +++ PVE/Storage.pm 2022-04-08 09:17:23.457073570 +0200 > @@ -412,6 +412,15 @@ >      return $plugin->get_subdir($scfg, 'iso'); >  } >   > +sub get_snippet_dir { > +    my ($cfg, $storeid) = @_; > + > +    my $scfg = storage_config($cfg, $storeid); > +    my $plugin = PVE::Storage::Plugin->lookup($scfg->{type}); > + > +    return $plugin->get_subdir($scfg, 'snippets'); > +} > + >  sub get_vztmpl_dir { >      my ($cfg, $storeid) = @_; >   > root@test:/usr/share/perl5# diff -u ./PVE/API2/Storage/Status.pm{- > orig,} > --- ./PVE/API2/Storage/Status.pm-orig 2022-04-08 > 09:15:43.883836880 +0200 > +++ ./PVE/API2/Storage/Status.pm 2022-04-08 10:23:43.914401204 > +0200 > @@ -381,7 +381,7 @@ >       content => { >   description => "Content type.", >   type => 'string', format => 'pve-storage-content', > - enum => ['iso', 'vztmpl'], > + enum => ['iso', 'vztmpl', 'snippets'], >       }, >       filename => { >   description => "The name of the file to create. > Caution: This will be normalized!", > @@ -446,8 +446,10 @@ >   raise_param_exc({ filename => "wrong file extension" > }); >       } >       $path = PVE::Storage::get_vztmpl_dir($cfg, $param- > >{storage}); > - } else { > -     raise_param_exc({ content => "upload content type > '$content' not allowed" }); > + } elsif ($content eq 'snippets') { > +     $path = PVE::Storage::get_snippet_dir($cfg, $param- > >{storage}); > +        } else { > +            raise_param_exc({ content => "upload content type > '$content' not allowed" }); >   } >   >   die "storage '$param->{storage}' does not support '$content' > content\n" > @@ -564,7 +566,7 @@ >       content => { >   description => "Content type.", # TODO: could be > optional & detected in most cases >   type => 'string', format => 'pve-storage-content', > - enum => ['iso', 'vztmpl'], > + enum => ['iso', 'vztmpl', 'snippets'], >       }, >       filename => { >   description => "The name of the file to create. > Caution: This will be normalized!", > @@ -627,6 +629,8 @@ >   raise_param_exc({ filename => "wrong file extension" > }); >       } >       $path = PVE::Storage::get_vztmpl_dir($cfg, $storage); > + } elsif ($content eq 'snippets') { > +     $path = PVE::Storage::get_snippet_dir($cfg, > $storage);     >   } else { >       raise_param_exc({ content => "upload content-type > '$content' is not allowed" }); >   } > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel