[pve-devel] [PATCH manager storage 0/2] fix #3716: allow https proxy for URL download

[pve-devel] [PATCH manager storage 0/2] fix #3716: allow https proxy for URL download

[Oguz Bektas]

we just reuse 'http_proxy' from datacenter.cfg

 pve-storage:

 Oguz Bektas (1):
  download-url: use https_proxy from datacenter.cfg (reuse http_proxy)

 PVE/API2/Storage/Status.pm | 1 +
 1 file changed, 1 insertion(+)

 pve-manager:

 Oguz Bektas (1):
  set https_proxy to http_proxy for querying url metadata

 PVE/API2/Nodes.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.30.2

[pve-devel] [PATCH storage 1/2] download-url: reuse http_proxy from datacenter.cfg for https

[Oguz Bektas]

$ tail -f /var/log/squid/access.log
...
1636466926.415  42386 127.0.0.1 TCP_TUNNEL/200 557422779 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -


Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
---
 PVE/API2/Storage/Status.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
index 02c970f..8eda39e 100644
--- a/PVE/API2/Storage/Status.pm
+++ b/PVE/API2/Storage/Status.pm
@@ -639,6 +639,7 @@ __PACKAGE__->register_method({
 	    hash_required => 0,
 	    verify_certificates => $param->{'verify-certificates'} // 1,
 	    http_proxy => $dccfg->{http_proxy},
+	    https_proxy => $dccfg->{http_proxy},
 	};
 
 	my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm'};
-- 
2.30.2

[pve-devel] [PATCH manager 2/2] set https_proxy to http_proxy for querying url metadata

[Oguz Bektas]

we can reuse the same 'http_proxy' from datacenter.cfg file.
requests made this way go via a http 'CONNECT'.

$ tail -f /var/log/squid/access.log
...
1636466841.211    463 127.0.0.1 TCP_TUNNEL/200 4717 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -
1636466841.696    481 127.0.0.1 TCP_TUNNEL/200 4725 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -
1636466842.179    480 127.0.0.1 TCP_TUNNEL/200 4783 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -


Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
---
 PVE/API2/Nodes.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
index 565cbccc..1e13285c 100644
--- a/PVE/API2/Nodes.pm
+++ b/PVE/API2/Nodes.pm
@@ -1548,7 +1548,7 @@ __PACKAGE__->register_method({
 
 	my $dccfg = PVE::Cluster::cfs_read_file('datacenter.cfg');
 	if ($dccfg->{http_proxy}) {
-	    $ua->proxy('http', $dccfg->{http_proxy});
+	    $ua->proxy("['http', 'https']", $dccfg->{http_proxy});
 	}
 
 	my $verify = $param->{'verify-certificates'} // 1;
-- 
2.30.2

Re: [pve-devel] [PATCH storage 1/2] download-url: reuse http_proxy from datacenter.cfg for https

[Dominik Csapak]

LGTM and works :)

On 11/9/21 15:13, Oguz Bektas wrote:
> $ tail -f /var/log/squid/access.log
> ...
> 1636466926.415  42386 127.0.0.1 TCP_TUNNEL/200 557422779 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -
> 
> 
> Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
> ---
>   PVE/API2/Storage/Status.pm | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
> index 02c970f..8eda39e 100644
> --- a/PVE/API2/Storage/Status.pm
> +++ b/PVE/API2/Storage/Status.pm
> @@ -639,6 +639,7 @@ __PACKAGE__->register_method({
>   	    hash_required => 0,
>   	    verify_certificates => $param->{'verify-certificates'} // 1,
>   	    http_proxy => $dccfg->{http_proxy},
> +	    https_proxy => $dccfg->{http_proxy},
>   	};
>   
>   	my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm'};
> 

Re: [pve-devel] [PATCH manager 2/2] set https_proxy to http_proxy for querying url metadata

[Dominik Csapak]

this sadly does not work, comment inline


On 11/9/21 15:13, Oguz Bektas wrote:
> we can reuse the same 'http_proxy' from datacenter.cfg file.
> requests made this way go via a http 'CONNECT'.
> 
> $ tail -f /var/log/squid/access.log
> ...
> 1636466841.211    463 127.0.0.1 TCP_TUNNEL/200 4717 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -
> 1636466841.696    481 127.0.0.1 TCP_TUNNEL/200 4725 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -
> 1636466842.179    480 127.0.0.1 TCP_TUNNEL/200 4783 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -
> 
> 
> Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
> ---
>   PVE/API2/Nodes.pm | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
> index 565cbccc..1e13285c 100644
> --- a/PVE/API2/Nodes.pm
> +++ b/PVE/API2/Nodes.pm
> @@ -1548,7 +1548,7 @@ __PACKAGE__->register_method({
>   
>   	my $dccfg = PVE::Cluster::cfs_read_file('datacenter.cfg');
>   	if ($dccfg->{http_proxy}) {
> -	    $ua->proxy('http', $dccfg->{http_proxy});
> +	    $ua->proxy("['http', 'https']", $dccfg->{http_proxy});

the protocols must be an arrayref, not a string so

$ua->proxy(['http', 'https'], ...)

would be correct

>   	}
>   
>   	my $verify = $param->{'verify-certificates'} // 1;
> 

Re: [pve-devel] [PATCH manager 2/2] set https_proxy to http_proxy for querying url metadata

[Oguz Bektas]

On Thu, Nov 25, 2021 at 02:34:54PM +0100, Dominik Csapak wrote:
> this sadly does not work, comment inline
> 
ah thanks... i guess i forgot to restart the daemon when testing. sent a
v2 now!

Re: [pve-devel] [PATCH storage 1/2] download-url: reuse http_proxy from datacenter.cfg for https

[Thomas Lamprecht]

On 25.11.21 14:34, Dominik Csapak wrote:
> LGTM and works :)
> 

in general has the same issue as the ACME one from Stoiko, namely:
The original http_proxy was always for external resources (our repos/appliances,
subscription checks), but this and the ACME ones aren't necesarrily external, and
proxying them may break some stuff (not all enterprise setups have control over the
proxy to make it differ between internal/external resources) or be just undesired.

What I'm missing on this and the acme patch is to actually step back and think
proxying in PVE/PMG through, what are the different use cases, how can they be
grouped sensible and exposed to the admin. At leas acknowledging something like
that in the commit message and giving some reasons about why that drawback is
accepted for now.

I mean, Stoiko at least made it a per-acme-plugin decision if something should get
proxied through the datacenter configured proxy or not, but one may want to have
different too (albeit blowing it up per single smallest request-type is surely overkill).

A https variant could be interesting too.

One could imagine a format string like (disclaimer, made up on the spot):

proxy: http=<>,https=<>,apply-on=<all|[base|acme|template-downloads]

(<base> would be the original repo/appliances/subscriber coverage)


> On 11/9/21 15:13, Oguz Bektas wrote:
>> $ tail -f /var/log/squid/access.log
>> ...
>> 1636466926.415  42386 127.0.0.1 TCP_TUNNEL/200 557422779 CONNECT fedorapeople.org:443 - HIER_DIRECT/152.19.134.199 -
>>
>>
>> Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
>> ---
>>   PVE/API2/Storage/Status.pm | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
>> index 02c970f..8eda39e 100644
>> --- a/PVE/API2/Storage/Status.pm
>> +++ b/PVE/API2/Storage/Status.pm
>> @@ -639,6 +639,7 @@ __PACKAGE__->register_method({
>>           hash_required => 0,
>>           verify_certificates => $param->{'verify-certificates'} // 1,
>>           http_proxy => $dccfg->{http_proxy},
>> +        https_proxy => $dccfg->{http_proxy},
>>       };
>>         my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm'};
>>

Re: [pve-devel] [PATCH manager 2/2] set https_proxy to http_proxy for querying url metadata

[Thomas Lamprecht]

On 25.11.21 14:49, Oguz Bektas wrote:
> On Thu, Nov 25, 2021 at 02:34:54PM +0100, Dominik Csapak wrote:
>> this sadly does not work, comment inline
>>
> ah thanks... i guess i forgot to restart the daemon when testing. sent a
> v2 now!
> 

So the squid access logs from the commit message are bogus?

Re: [pve-devel] [PATCH manager 2/2] set https_proxy to http_proxy for querying url metadata

[Oguz Bektas]

On Thu, Nov 25, 2021 at 03:07:58PM +0100, Thomas Lamprecht wrote:
> On 25.11.21 14:49, Oguz Bektas wrote:
> > On Thu, Nov 25, 2021 at 02:34:54PM +0100, Dominik Csapak wrote:
> >> this sadly does not work, comment inline
> >>
> > ah thanks... i guess i forgot to restart the daemon when testing. sent a
> > v2 now!
> > 
> 
> So the squid access logs from the commit message are bogus?
> 

no... the original one i had was with two lines instead of the array so
it worked properly. i just forgot to restart pveproxy after changing and didnt notice.

Re: [pve-devel] [PATCH storage 1/2] download-url: reuse http_proxy from datacenter.cfg for https

[Dominik Csapak]

On 11/25/21 15:06, Thomas Lamprecht wrote:
> On 25.11.21 14:34, Dominik Csapak wrote:
>> LGTM and works :)
>>
> 
> in general has the same issue as the ACME one from Stoiko, namely:
> The original http_proxy was always for external resources (our repos/appliances,
> subscription checks), but this and the ACME ones aren't necesarrily external, and
> proxying them may break some stuff (not all enterprise setups have control over the
> proxy to make it differ between internal/external resources) or be just undesired.
> 
> What I'm missing on this and the acme patch is to actually step back and think
> proxying in PVE/PMG through, what are the different use cases, how can they be
> grouped sensible and exposed to the admin. At leas acknowledging something like
> that in the commit message and giving some reasons about why that drawback is
> accepted for now.
> 
> I mean, Stoiko at least made it a per-acme-plugin decision if something should get
> proxied through the datacenter configured proxy or not, but one may want to have
> different too (albeit blowing it up per single smallest request-type is surely overkill).
> 
> A https variant could be interesting too.
> 
> One could imagine a format string like (disclaimer, made up on the spot):
> 
> proxy: http=<>,https=<>,apply-on=<all|[base|acme|template-downloads]
> 
> (<base> would be the original repo/appliances/subscriber coverage)
> 
>

just to note, i am not disagreeing with you but a small comment to this
patch nonetheless:

imho the current state is rather broken, for http urls it uses the proxy 
but not for https (isos are often hosted on https for now, e.g. debians)
ans since at least one person[0] expected it to use the proxy in any
case, i'd argue that for now this would be ok

yes a more general approach with specific uses/allow/blocklist (however
we want to implement this) would be better, but can still
be done after this

0: https://bugzilla.proxmox.com/show_bug.cgi?id=3716