From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 0D0C71FF162 for <inbox@lore.proxmox.com>; Sat, 5 Apr 2025 18:29:25 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 36D43E385; Sat, 5 Apr 2025 18:29:23 +0200 (CEST) Message-ID: <3fcc3db8-148f-488f-bead-ea70a7431156@proxmox.com> Date: Sat, 5 Apr 2025 18:28:48 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Friedrich Weber <f.weber@proxmox.com> References: <20250326105108.34911-1-f.weber@proxmox.com> Content-Language: en-GB, de-AT From: Thomas Lamprecht <t.lamprecht@proxmox.com> Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <20250326105108.34911-1-f.weber@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL -0.039 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [status.pm, nodes.pm, proxmox.com] Subject: [pve-devel] applied-series: [PATCH manager/storage 0/2] fix #3716: allow downloading iso/vztmpl/ova via https in proxied environments X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com> Am 26.03.25 um 11:51 schrieb Friedrich Weber: > A user in enterprise support reported (and users also reported elsewhere [1] > [2]) that ISO downloads via https currently do not work in environments using a > proxy configured via the datacenter option `http_proxy`, if the connection to > the ISO repository needs to go via the proxy. Proxied ISO downloads via http > work. OVA and VZ template downloads are also affected. > > The reason is that > > - when querying the metadata via LWP, the proxy is only set for the `http` > scheme, not `https` > > - when spawning wget to download the ISO, only the `http_proxy` environment > variable is set, not `https_proxy`, so wget will not use a proxy for the https > connection > > Hence, neither operation uses the proxy for https. If the node cannot reach the > destination without the proxy, both operations time out. > > Fix this by > > - patch 1: setting the `http_proxy` from the datacenter config also for the > https scheme when querying the metadata via LWP > > - patch 2: passing the `https_proxy` environment variable to the wget command, > setting it to the `http_proxy` from the datacenter config > > Tested by running squid in a container, and setting up the firewall to drop > outgoing traffic from the PVE to everything but the proxy. Running > tcpconnect-bpfcc from bpfcc-tools helps for tracing the destination of the > http/https connections. > > Maximiliano and I discussed this and looked into earlier iterations on this. > > When a similar series was initially sent in 2021 [3], Thomas raised the concern > [4] that our proxy settings should be more fine-grained and allow the user to > differentiate between resources that should be proxied and that should not be > proxied. For example, ISO repositories may be external (and thus may only be > reachable via proxying) or internal (and thus may not be reachable via the > proxy). Same for ACME endpoints. One idea was to group http requests issued by > our stack into categories (such as `base`, `acme`, `template`) and allow the > proxy setting to only apply to certain categories. I agree that something like > this sounds like a useful feature, and one user also requested [5] something > along these lines. > > However, Maximiliano and I would argue this concern is orthogonal to the issue > fixed by this patch. If a user has configured `http_proxy`, having the ISO > download work via http and fail via https is inconsistent and thus confusing > (see also Dominik's post [6] from back then). It might even nudge users into > using http instead (which can still give the same integrity guarantees if they > retrieve the checksum via https and compare them, but this is easy to get > wrong). We'd propose we use the `http_proxy` for both https and https for now, > and can still look into the categorization feature #5420 later. I agree with that it being orthogonal, so while it's not ideal it still improves the status quo. > > Other places in our stack also use the `http_proxy` datacenter option for https > connections, e.g. the ones that use proxmox_http::HttpClient with ProxyConfig > such as with the notification system's webhook endpoint. > > One argument against this patch is that it breaks backwards compatibility: > Existing setups with `http_proxy` and an *internal* ISO repository from which > they download via https will break. If this is a concern, I'd suggest we wait > for PVE 9 to apply this. It is a concern, but not sure if delaying this to a major release is of any help if there is then still no way to workaround that without disabling and then re-setting the proxy setting before/after such an operation. > What do you think? I've seen published papers that were less elaborate and less convincing ;-) > > FTR, there is some overlap with a patch series by Hannes Laimer [7] but that > one only concerned the `query-url-metadata` and the `apl_download` endpoints > (for downloading appliance templates), not the ISO download. > > [1] https://bugzilla.proxmox.com/show_bug.cgi?id=3716 > [2] https://bugzilla.proxmox.com/show_bug.cgi?id=5420#c2 > [3] https://lore.proxmox.com/pve-devel/20211109141359.990235-1-o.bektas@proxmox.com/ > [4] https://lore.proxmox.com/pve-devel/42391428-bd80-2d55-5cb6-7c8ecd97a3a8@proxmox.com/ > [5] https://bugzilla.proxmox.com/show_bug.cgi?id=5420#c0 > [6] https://lore.proxmox.com/pve-devel/a03631a3-fe78-7f6f-137d-7ee6fdf8f9ed@proxmox.com/ > [7] https://git.proxmox.com/?p=proxmox.git;a=blob;f=proxmox-notify/src/endpoints/webhook.rs;h=34dbac5488;hb=7abd2da759d#l266 > [8] https://lore.proxmox.com/pve-devel/20240308123535.1500-1-h.laimer@proxmox.com/ > > Co-authored-by: Maximiliano Sandoval <m.sandoval@proxmox.com> > > storage: > > Friedrich Weber (1): > fix #3716: api: download from url: use proxy option for https > > src/PVE/API2/Storage/Status.pm | 1 + > 1 file changed, 1 insertion(+) > > > manager: > > Friedrich Weber (1): > fix #3716: api: nodes: query metadata: use proxy option for https > > PVE/API2/Nodes.pm | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > > Summary over all repositories: > 2 files changed, 2 insertions(+), 1 deletions(-) > applied series, thanks for bringing this up again and adding strong arguments! _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel