* [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http
@ 2021-05-14 13:44 Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
` (20 more replies)
0 siblings, 21 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
a first batch of factoring out HTTP helpers and client code into
a new proxmox_http crate living in the proxmox workspace. the
full-fledged API client ist still in proxmox-backup as it is entagled
with Userid/Authid, and we need to decide whether to move those into
proxmox as well or make the API client generic over some sort of
credentials trait/type/..
proxmox:
Fabian Grünbichler (13):
proxmox: add missing +router -> futures dep
meta: add empty proxmox-http sub-crate
http: takeover websocket feature from proxmox
http: make clippy happy
proxmox: takeover socket helper from proxmox_backup
http: takeover MaybeTlsStream from proxmox_backup
http: takeover build_authority helper from proxmox_backup
http: takeover ProxyConfig from proxmox_backup
http: takeover tools::http from proxmox_backup
http: takeover simple HTTP client from proxmox_backup
http: make clippy happy
http: rustfmt
http: track d/control
21 files changed, 900 insertions(+), 43 deletions(-)
proxmox-backup:
Fabian Grünbichler (8):
move websocket to new 'proxmox_http' crate
refactor: move socket helper to proxmox crate
move MaybeTlsStream wrapper to proxmox_http
HttpsConnector: make keepalive configurable
move ProxyConfig to proxmox_http
move tools::http to proxmox_http
SimpleHttp: factor out product-specific bits
move SimpleHttp to proxmox_http
15 files changed, 58 insertions(+), 585 deletions(-)
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-17 8:22 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate Fabian Grünbichler
` (19 subsequent siblings)
20 siblings, 1 reply; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
this is shadowed by the websocket feature being enabled by default.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
fixes issue uncovered by next patch
proxmox/Cargo.toml | 2 +-
proxmox/debian/control | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/proxmox/Cargo.toml b/proxmox/Cargo.toml
index c04c105..da46e74 100644
--- a/proxmox/Cargo.toml
+++ b/proxmox/Cargo.toml
@@ -57,7 +57,7 @@ sortable-macro = ["proxmox-sortable-macro"]
api-macro = ["proxmox-api-macro"]
test-harness = []
cli = [ "router", "hyper", "tokio" ]
-router = [ "hyper", "tokio" ]
+router = [ "futures", "hyper", "tokio" ]
websocket = [ "futures", "hyper", "openssl", "tokio/sync", "tokio/io-util", "openssl" ]
tfa = [ "openssl" ]
u2f = [ "base32" ]
diff --git a/proxmox/debian/control b/proxmox/debian/control
index da55b88..9091cb8 100644
--- a/proxmox/debian/control
+++ b/proxmox/debian/control
@@ -267,6 +267,7 @@ Multi-Arch: same
Depends:
${misc:Depends},
librust-proxmox-dev (= ${binary:Version}),
+ librust-futures-0.3+default-dev,
librust-hyper-0.14+default-dev,
librust-hyper-0.14+full-dev,
librust-tokio-1+default-dev
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox Fabian Grünbichler
` (18 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Makefile | 2 +-
Cargo.toml | 1 +
proxmox-http/Cargo.toml | 20 +++
proxmox-http/debian/changelog | 5 +
proxmox-http/debian/control | 230 ++++++++++++++++++++++++++++++
proxmox-http/debian/copyright | 16 +++
proxmox-http/debian/debcargo.toml | 7 +
proxmox-http/src/lib.rs | 0
8 files changed, 280 insertions(+), 1 deletion(-)
create mode 100644 proxmox-http/Cargo.toml
create mode 100644 proxmox-http/debian/changelog
create mode 100644 proxmox-http/debian/control
create mode 100644 proxmox-http/debian/copyright
create mode 100644 proxmox-http/debian/debcargo.toml
create mode 100644 proxmox-http/src/lib.rs
diff --git a/Makefile b/Makefile
index ebaf113..4f74539 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
# Shortcut for common operations:
-CRATES=proxmox proxmox-api-macro proxmox-sortable-macro
+CRATES=proxmox proxmox-api-macro proxmox-http proxmox-sortable-macro
# By default we just run checks:
.PHONY: all
diff --git a/Cargo.toml b/Cargo.toml
index d6dc7b4..12e832d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -2,6 +2,7 @@
members = [
"proxmox",
"proxmox-api-macro",
+ "proxmox-http",
"proxmox-sortable-macro",
]
exclude = [
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
new file mode 100644
index 0000000..9c2d3c5
--- /dev/null
+++ b/proxmox-http/Cargo.toml
@@ -0,0 +1,20 @@
+[package]
+name = "proxmox-http"
+edition = "2018"
+version = "0.1.0"
+authors = [
+ "Dietmar Maurer <dietmar@proxmox.com>",
+ "Wolfgang Bumiller <w.bumiller@proxmox.com>",
+]
+license = "AGPL-3"
+description = "Proxmox HTTP library"
+
+exclude = [ "debian" ]
+
+[dependencies]
+
+[features]
+default = []
+
+client = []
+websocket = []
diff --git a/proxmox-http/debian/changelog b/proxmox-http/debian/changelog
new file mode 100644
index 0000000..2ef333f
--- /dev/null
+++ b/proxmox-http/debian/changelog
@@ -0,0 +1,5 @@
+rust-proxmox-http (0.1.0-1) unstable; urgency=medium
+
+ * refactor code from proxmox-backup and proxmox into own HTTP crate
+
+ -- Proxmox Support Team <support@proxmox.com> Tue, 11 May 2021 10:50:29 +0200
diff --git a/proxmox-http/debian/control b/proxmox-http/debian/control
new file mode 100644
index 0000000..f010d6e
--- /dev/null
+++ b/proxmox-http/debian/control
@@ -0,0 +1,230 @@
+Source: rust-proxmox-http
+Section: rust
+Priority: optional
+Build-Depends: debhelper (>= 11),
+ dh-cargo (>= 18),
+ cargo:native <!nocheck>,
+ rustc:native <!nocheck>,
+ libstd-rust-dev <!nocheck>,
+ librust-anyhow-1+default-dev <!nocheck>
+Maintainer: Proxmox Support Team <support@proxmox.com>
+Standards-Version: 4.4.1
+Vcs-Git: git://git.proxmox.com/git/proxmox.git
+Vcs-Browser: https://git.proxmox.com/?p=proxmox.git
+
+Package: librust-proxmox-http-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-anyhow-1+default-dev
+Suggests:
+ librust-proxmox-http+base64-dev (= ${binary:Version}),
+ librust-proxmox-http+client-dev (= ${binary:Version}),
+ librust-proxmox-http+futures-dev (= ${binary:Version}),
+ librust-proxmox-http+http-dev (= ${binary:Version}),
+ librust-proxmox-http+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http+hyper-dev (= ${binary:Version}),
+ librust-proxmox-http+openssl-dev (= ${binary:Version}),
+ librust-proxmox-http+proxmox-dev (= ${binary:Version}),
+ librust-proxmox-http+tokio-dev (= ${binary:Version}),
+ librust-proxmox-http+tokio-openssl-dev (= ${binary:Version}),
+ librust-proxmox-http+websocket-dev (= ${binary:Version})
+Provides:
+ librust-proxmox-http+default-dev (= ${binary:Version}),
+ librust-proxmox-http-0-dev (= ${binary:Version}),
+ librust-proxmox-http-0+default-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+default-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+default-dev (= ${binary:Version})
+Description: Proxmox HTTP library - Rust source code
+ This package contains the source for the Rust proxmox-http crate, packaged by
+ debcargo for use with cargo and dh-cargo.
+
+Package: librust-proxmox-http+base64-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-base64-0.12+default-dev
+Provides:
+ librust-proxmox-http-0+base64-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+base64-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+base64-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "base64"
+ This metapackage enables feature "base64" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+client-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-proxmox-http+http-helpers-dev (= ${binary:Version}),
+ librust-futures-0.3+default-dev,
+ librust-openssl-0.10+default-dev
+Provides:
+ librust-proxmox-http-0+client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+client-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "client"
+ This metapackage enables feature "client" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+futures-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-futures-0.3+default-dev
+Provides:
+ librust-proxmox-http-0+futures-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+futures-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+futures-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "futures"
+ This metapackage enables feature "futures" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+http-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-http-0.2+default-dev
+Provides:
+ librust-proxmox-http-0+http-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+http-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+http-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "http"
+ This metapackage enables feature "http" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+http-helpers-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-base64-0.12+default-dev,
+ librust-http-0.2+default-dev,
+ librust-hyper-0.14+default-dev,
+ librust-hyper-0.14+full-dev,
+ librust-proxmox-0.11-dev (>= 0.11.3-~~),
+ librust-tokio-1+io-util-dev,
+ librust-tokio-openssl-0.6+default-dev (>= 0.6.1-~~)
+Provides:
+ librust-proxmox-http+simple-client-dev (= ${binary:Version}),
+ librust-proxmox-http-0+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http-0+simple-client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+simple-client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+simple-client-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "http-helpers" and 1 more
+ This metapackage enables feature "http-helpers" for the Rust proxmox-http
+ crate, by pulling in any additional dependencies needed by that feature.
+ .
+ Additionally, this package also provides the "simple-client" feature.
+
+Package: librust-proxmox-http+hyper-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-hyper-0.14+default-dev,
+ librust-hyper-0.14+full-dev
+Provides:
+ librust-proxmox-http-0+hyper-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+hyper-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+hyper-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "hyper"
+ This metapackage enables feature "hyper" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+openssl-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-openssl-0.10+default-dev
+Provides:
+ librust-proxmox-http-0+openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+openssl-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "openssl"
+ This metapackage enables feature "openssl" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+proxmox-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-proxmox-0.11-dev (>= 0.11.3-~~)
+Provides:
+ librust-proxmox-http-0+proxmox-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+proxmox-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+proxmox-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "proxmox"
+ This metapackage enables feature "proxmox" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+tokio-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-tokio-1+default-dev
+Provides:
+ librust-proxmox-http-0+tokio-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+tokio-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+tokio-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "tokio"
+ This metapackage enables feature "tokio" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+tokio-openssl-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-tokio-openssl-0.6+default-dev (>= 0.6.1-~~)
+Provides:
+ librust-proxmox-http-0+tokio-openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+tokio-openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+tokio-openssl-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "tokio-openssl"
+ This metapackage enables feature "tokio-openssl" for the Rust proxmox-http
+ crate, by pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+websocket-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-base64-0.12+default-dev,
+ librust-futures-0.3+default-dev,
+ librust-hyper-0.14+default-dev,
+ librust-hyper-0.14+full-dev,
+ librust-openssl-0.10+default-dev,
+ librust-proxmox-0.11+tokio-dev (>= 0.11.3-~~),
+ librust-tokio-1+io-util-dev,
+ librust-tokio-1+sync-dev
+Provides:
+ librust-proxmox-http-0+websocket-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+websocket-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+websocket-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "websocket"
+ This metapackage enables feature "websocket" for the Rust proxmox-http crate,
+ by pulling in any additional dependencies needed by that feature.
diff --git a/proxmox-http/debian/copyright b/proxmox-http/debian/copyright
new file mode 100644
index 0000000..2c91d38
--- /dev/null
+++ b/proxmox-http/debian/copyright
@@ -0,0 +1,16 @@
+Copyright (C) 2019,2020 Proxmox Server Solutions GmbH
+
+This software is written by Proxmox Server Solutions GmbH <support@proxmox.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
diff --git a/proxmox-http/debian/debcargo.toml b/proxmox-http/debian/debcargo.toml
new file mode 100644
index 0000000..b7864cd
--- /dev/null
+++ b/proxmox-http/debian/debcargo.toml
@@ -0,0 +1,7 @@
+overlay = "."
+crate_src_path = ".."
+maintainer = "Proxmox Support Team <support@proxmox.com>"
+
+[source]
+vcs_git = "git://git.proxmox.com/git/proxmox.git"
+vcs_browser = "https://git.proxmox.com/?p=proxmox.git"
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
new file mode 100644
index 0000000..e69de29
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy Fabian Grünbichler
` (17 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
adapted: use statements for proxmox::*, use statements for doctests
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
BREAKING change! requires corresponding update in proxmox-backup
proxmox-http/Cargo.toml | 10 ++++++-
proxmox-http/src/lib.rs | 2 ++
.../src/websocket/mod.rs | 14 ++++-----
proxmox/Cargo.toml | 3 +-
proxmox/debian/control | 29 ++-----------------
proxmox/src/tools/mod.rs | 3 --
6 files changed, 21 insertions(+), 40 deletions(-)
rename proxmox/src/tools/websocket.rs => proxmox-http/src/websocket/mod.rs (98%)
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 9c2d3c5..641ace2 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -12,9 +12,17 @@ description = "Proxmox HTTP library"
exclude = [ "debian" ]
[dependencies]
+anyhow = "1.0"
+base64 = { version = "0.12", optional = true }
+futures = { version = "0.3", optional = true }
+hyper = { version = "0.14", features = [ "full" ], optional = true }
+openssl = { version = "0.10", optional = true }
+tokio = { version = "1.0", features = [], optional = true }
+
+proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-features = false }
[features]
default = []
client = []
-websocket = []
+websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
index e69de29..31e44af 100644
--- a/proxmox-http/src/lib.rs
+++ b/proxmox-http/src/lib.rs
@@ -0,0 +1,2 @@
+#[cfg(feature = "websocket")]
+pub mod websocket;
diff --git a/proxmox/src/tools/websocket.rs b/proxmox-http/src/websocket/mod.rs
similarity index 98%
rename from proxmox/src/tools/websocket.rs
rename to proxmox-http/src/websocket/mod.rs
index 83679b6..aac7304 100644
--- a/proxmox/src/tools/websocket.rs
+++ b/proxmox-http/src/websocket/mod.rs
@@ -22,8 +22,8 @@ use tokio::sync::mpsc;
use futures::future::FutureExt;
use futures::ready;
-use crate::sys::error::io_err_other;
-use crate::tools::byte_buffer::ByteBuffer;
+use proxmox::sys::error::io_err_other;
+use proxmox::tools::byte_buffer::ByteBuffer;
// see RFC6455 section 7.4.1
#[derive(Debug, Clone, Copy)]
@@ -146,7 +146,7 @@ fn mask_bytes(mask: Option<[u8; 4]>, data: &mut [u8]) {
///
/// A normal Frame
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let data = vec![0,1,2,3,4];
@@ -159,7 +159,7 @@ fn mask_bytes(mask: Option<[u8; 4]>, data: &mut [u8]) {
///
/// A masked Frame
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let data = vec![0,1,2,3,4];
@@ -172,7 +172,7 @@ fn mask_bytes(mask: Option<[u8; 4]>, data: &mut [u8]) {
///
/// A ping Frame
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let data = vec![0,1,2,3,4];
@@ -233,7 +233,7 @@ pub fn create_frame(
///
/// Example usage:
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # use tokio::io::{AsyncWrite, AsyncWriteExt};
/// async fn code<I: AsyncWrite + Unpin>(writer: I) -> io::Result<()> {
@@ -352,7 +352,7 @@ impl FrameHeader {
///
/// Example:
/// ```
- /// # use proxmox::tools::websocket::*;
+ /// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let frame = create_frame(None, &[0,1,2,3], OpCode::Ping)?;
diff --git a/proxmox/Cargo.toml b/proxmox/Cargo.toml
index da46e74..5698969 100644
--- a/proxmox/Cargo.toml
+++ b/proxmox/Cargo.toml
@@ -50,7 +50,7 @@ proxmox-api-macro = { path = "../proxmox-api-macro", optional = true, version =
proxmox-sortable-macro = { path = "../proxmox-sortable-macro", optional = true, version = "0.1.1" }
[features]
-default = [ "cli", "router", "tfa", "u2f", "websocket" ]
+default = [ "cli", "router", "tfa", "u2f" ]
sortable-macro = ["proxmox-sortable-macro"]
# api:
@@ -58,7 +58,6 @@ api-macro = ["proxmox-api-macro"]
test-harness = []
cli = [ "router", "hyper", "tokio" ]
router = [ "futures", "hyper", "tokio" ]
-websocket = [ "futures", "hyper", "openssl", "tokio/sync", "tokio/io-util", "openssl" ]
tfa = [ "openssl" ]
u2f = [ "base32" ]
diff --git a/proxmox/debian/control b/proxmox/debian/control
index 9091cb8..0ba2bc9 100644
--- a/proxmox/debian/control
+++ b/proxmox/debian/control
@@ -28,15 +28,12 @@ Build-Depends: debhelper (>= 11),
librust-serde-json-1+default-dev <!nocheck>,
librust-textwrap-0.11+default-dev <!nocheck>,
librust-tokio-1+default-dev <!nocheck>,
- librust-tokio-1+io-util-dev <!nocheck>,
- librust-tokio-1+sync-dev <!nocheck>,
librust-url-2+default-dev (>= 2.1-~~) <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
- uuid-dev <!nocheck>,
uuid-dev <!nocheck>
Maintainer: Proxmox Support Team <support@proxmox.com>
Standards-Version: 4.4.1
@@ -79,8 +76,7 @@ Suggests:
librust-proxmox+proxmox-sortable-macro-dev (= ${binary:Version}),
librust-proxmox+router-dev (= ${binary:Version}),
librust-proxmox+tokio-dev (= ${binary:Version}),
- librust-proxmox+tokio-stream-dev (= ${binary:Version}),
- librust-proxmox+websocket-dev (= ${binary:Version})
+ librust-proxmox+tokio-stream-dev (= ${binary:Version})
Provides:
librust-proxmox+test-harness-dev (= ${binary:Version}),
librust-proxmox-0-dev (= ${binary:Version}),
@@ -162,8 +158,7 @@ Depends:
librust-proxmox+cli-dev (= ${binary:Version}),
librust-proxmox+router-dev (= ${binary:Version}),
librust-proxmox+tfa-dev (= ${binary:Version}),
- librust-proxmox+u2f-dev (= ${binary:Version}),
- librust-proxmox+websocket-dev (= ${binary:Version})
+ librust-proxmox+u2f-dev (= ${binary:Version})
Provides:
librust-proxmox-0+default-dev (= ${binary:Version}),
librust-proxmox-0.11+default-dev (= ${binary:Version}),
@@ -308,23 +303,3 @@ Provides:
Description: Proxmox library - feature "tokio-stream"
This metapackage enables feature "tokio-stream" for the Rust proxmox crate, by
pulling in any additional dependencies needed by that feature.
-
-Package: librust-proxmox+websocket-dev
-Architecture: any
-Multi-Arch: same
-Depends:
- ${misc:Depends},
- librust-proxmox-dev (= ${binary:Version}),
- librust-futures-0.3+default-dev,
- librust-hyper-0.14+default-dev,
- librust-hyper-0.14+full-dev,
- librust-openssl-0.10+default-dev,
- librust-tokio-1+io-util-dev,
- librust-tokio-1+sync-dev
-Provides:
- librust-proxmox-0+websocket-dev (= ${binary:Version}),
- librust-proxmox-0.11+websocket-dev (= ${binary:Version}),
- librust-proxmox-0.11.4+websocket-dev (= ${binary:Version})
-Description: Proxmox library - feature "websocket"
- This metapackage enables feature "websocket" for the Rust proxmox crate, by
- pulling in any additional dependencies needed by that feature.
diff --git a/proxmox/src/tools/mod.rs b/proxmox/src/tools/mod.rs
index 45f46f9..b172b09 100644
--- a/proxmox/src/tools/mod.rs
+++ b/proxmox/src/tools/mod.rs
@@ -21,9 +21,6 @@ pub mod time;
pub mod uuid;
pub mod vec;
-#[cfg(feature = "websocket")]
-pub mod websocket;
-
#[cfg(feature = "tfa")]
pub mod tfa;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (2 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup Fabian Grünbichler
` (16 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/websocket/mod.rs | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/proxmox-http/src/websocket/mod.rs b/proxmox-http/src/websocket/mod.rs
index aac7304..b8f31dc 100644
--- a/proxmox-http/src/websocket/mod.rs
+++ b/proxmox-http/src/websocket/mod.rs
@@ -202,8 +202,7 @@ pub fn create_frame(
0b00000000
};
- let mut buf = Vec::new();
- buf.push(first_byte);
+ let mut buf = vec![first_byte];
if len < 126 {
buf.push(mask_bit | (len as u8));
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (3 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream " Fabian Grünbichler
` (15 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox/src/sys/linux/mod.rs | 1 +
proxmox/src/sys/linux/socket.rs | 21 +++++++++++++++++++++
2 files changed, 22 insertions(+)
create mode 100644 proxmox/src/sys/linux/socket.rs
diff --git a/proxmox/src/sys/linux/mod.rs b/proxmox/src/sys/linux/mod.rs
index ee616a3..ccd710c 100644
--- a/proxmox/src/sys/linux/mod.rs
+++ b/proxmox/src/sys/linux/mod.rs
@@ -6,6 +6,7 @@ pub mod magic;
pub mod pid;
pub mod procfs;
pub mod pty;
+pub mod socket;
pub mod tty;
/// Get pseudo random data (/dev/urandom)
diff --git a/proxmox/src/sys/linux/socket.rs b/proxmox/src/sys/linux/socket.rs
new file mode 100644
index 0000000..6c63c3c
--- /dev/null
+++ b/proxmox/src/sys/linux/socket.rs
@@ -0,0 +1,21 @@
+use std::os::unix::io::RawFd;
+
+use nix::sys::socket::sockopt::{KeepAlive, TcpKeepIdle};
+use nix::sys::socket::setsockopt;
+
+/// Set TCP keepalive time on a socket
+///
+/// See "man 7 tcp" for details.
+///
+/// The default on Linux is 7200 (2 hours) which is far too long for
+/// many of our use cases.
+pub fn set_tcp_keepalive(
+ socket_fd: RawFd,
+ tcp_keepalive_time: u32,
+) -> nix::Result<()> {
+
+ setsockopt(socket_fd, KeepAlive, &true)?;
+ setsockopt(socket_fd, TcpKeepIdle, &tcp_keepalive_time)?;
+
+ Ok(())
+}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (4 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper " Fabian Grünbichler
` (14 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
this is just a (rather HTTP specific) wrapper, so put it into a
'wrapper' module for now.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 4 +-
proxmox-http/src/http/mod.rs | 3 +
proxmox-http/src/http/wrapper.rs | 122 +++++++++++++++++++++++++++++++
proxmox-http/src/lib.rs | 3 +
4 files changed, 131 insertions(+), 1 deletion(-)
create mode 100644 proxmox-http/src/http/mod.rs
create mode 100644 proxmox-http/src/http/wrapper.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 641ace2..f1f53da 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -18,11 +18,13 @@ futures = { version = "0.3", optional = true }
hyper = { version = "0.14", features = [ "full" ], optional = true }
openssl = { version = "0.10", optional = true }
tokio = { version = "1.0", features = [], optional = true }
+tokio-openssl = { version = "0.6.1", optional = true }
proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-features = false }
[features]
default = []
-client = []
+client = [ "http-helpers" ]
+http-helpers = [ "hyper", "tokio/io-util", "tokio-openssl" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
new file mode 100644
index 0000000..09fa42f
--- /dev/null
+++ b/proxmox-http/src/http/mod.rs
@@ -0,0 +1,3 @@
+mod wrapper;
+
+pub use wrapper::MaybeTlsStream;
diff --git a/proxmox-http/src/http/wrapper.rs b/proxmox-http/src/http/wrapper.rs
new file mode 100644
index 0000000..3399b28
--- /dev/null
+++ b/proxmox-http/src/http/wrapper.rs
@@ -0,0 +1,122 @@
+use std::io;
+use std::pin::Pin;
+use std::task::{Context, Poll};
+
+use hyper::client::connect::{Connection, Connected};
+use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
+use tokio_openssl::SslStream;
+
+/// Asynchronous stream, possibly encrypted and proxied
+///
+/// Usefule for HTTP client implementations using hyper.
+pub enum MaybeTlsStream<S> {
+ Normal(S),
+ Proxied(S),
+ Secured(SslStream<S>),
+}
+
+impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
+ fn poll_read(
+ self: Pin<&mut Self>,
+ cx: &mut Context,
+ buf: &mut ReadBuf,
+ ) -> Poll<Result<(), io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_read(cx, buf)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_read(cx, buf)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_read(cx, buf)
+ }
+ }
+ }
+}
+
+impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
+ fn poll_write(
+ self: Pin<&mut Self>,
+ cx: &mut Context,
+ buf: &[u8],
+ ) -> Poll<Result<usize, io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_write(cx, buf)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_write(cx, buf)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_write(cx, buf)
+ }
+ }
+ }
+
+ fn poll_write_vectored(
+ self: Pin<&mut Self>,
+ cx: &mut Context<'_>,
+ bufs: &[io::IoSlice<'_>],
+ ) -> Poll<Result<usize, io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_write_vectored(cx, bufs)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_write_vectored(cx, bufs)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_write_vectored(cx, bufs)
+ }
+ }
+ }
+
+ fn is_write_vectored(&self) -> bool {
+ match self {
+ MaybeTlsStream::Normal(s) => s.is_write_vectored(),
+ MaybeTlsStream::Proxied(s) => s.is_write_vectored(),
+ MaybeTlsStream::Secured(s) => s.is_write_vectored(),
+ }
+ }
+
+ fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_flush(cx)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_flush(cx)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_flush(cx)
+ }
+ }
+ }
+
+ fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_shutdown(cx)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_shutdown(cx)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_shutdown(cx)
+ }
+ }
+ }
+}
+
+// we need this for the hyper http client
+impl <S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S>
+{
+ fn connected(&self) -> Connected {
+ match self {
+ MaybeTlsStream::Normal(s) => s.connected(),
+ MaybeTlsStream::Proxied(s) => s.connected().proxy(true),
+ MaybeTlsStream::Secured(s) => s.get_ref().connected(),
+ }
+ }
+}
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
index 31e44af..5e3f9ec 100644
--- a/proxmox-http/src/lib.rs
+++ b/proxmox-http/src/lib.rs
@@ -1,2 +1,5 @@
#[cfg(feature = "websocket")]
pub mod websocket;
+
+#[cfg(feature = "http-helpers")]
+pub mod http;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (5 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig " Fabian Grünbichler
` (13 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 3 ++-
proxmox-http/src/http/helpers.rs | 15 +++++++++++++++
proxmox-http/src/http/mod.rs | 2 ++
3 files changed, 19 insertions(+), 1 deletion(-)
create mode 100644 proxmox-http/src/http/helpers.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index f1f53da..6b2d8db 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -15,6 +15,7 @@ exclude = [ "debian" ]
anyhow = "1.0"
base64 = { version = "0.12", optional = true }
futures = { version = "0.3", optional = true }
+http = { version = "0.2", optional = true }
hyper = { version = "0.14", features = [ "full" ], optional = true }
openssl = { version = "0.10", optional = true }
tokio = { version = "1.0", features = [], optional = true }
@@ -26,5 +27,5 @@ proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-fe
default = []
client = [ "http-helpers" ]
-http-helpers = [ "hyper", "tokio/io-util", "tokio-openssl" ]
+http-helpers = [ "http", "hyper", "tokio/io-util", "tokio-openssl" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/helpers.rs b/proxmox-http/src/http/helpers.rs
new file mode 100644
index 0000000..3f663d2
--- /dev/null
+++ b/proxmox-http/src/http/helpers.rs
@@ -0,0 +1,15 @@
+use anyhow::Error;
+
+use http::uri::Authority;
+
+// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
+pub fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
+ let bytes = host.as_bytes();
+ let len = bytes.len();
+ let authority = if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len-1] != b']' {
+ format!("[{}]:{}", host, port).parse()?
+ } else {
+ format!("{}:{}", host, port).parse()?
+ };
+ Ok(authority)
+}
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
index 09fa42f..4960246 100644
--- a/proxmox-http/src/http/mod.rs
+++ b/proxmox-http/src/http/mod.rs
@@ -1,3 +1,5 @@
mod wrapper;
pub use wrapper::MaybeTlsStream;
+
+pub mod helpers;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (6 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http " Fabian Grünbichler
` (12 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
adapted to use moved build_authority helper.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 2 +-
proxmox-http/src/http/mod.rs | 4 +-
proxmox-http/src/http/proxy_config.rs | 84 +++++++++++++++++++++++++++
3 files changed, 88 insertions(+), 2 deletions(-)
create mode 100644 proxmox-http/src/http/proxy_config.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 6b2d8db..9c6fc35 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -27,5 +27,5 @@ proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-fe
default = []
client = [ "http-helpers" ]
-http-helpers = [ "http", "hyper", "tokio/io-util", "tokio-openssl" ]
+http-helpers = [ "base64", "http", "hyper", "tokio/io-util", "tokio-openssl", "proxmox" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
index 4960246..055648e 100644
--- a/proxmox-http/src/http/mod.rs
+++ b/proxmox-http/src/http/mod.rs
@@ -1,5 +1,7 @@
mod wrapper;
-
pub use wrapper::MaybeTlsStream;
pub mod helpers;
+
+mod proxy_config;
+pub use proxy_config::ProxyConfig;
diff --git a/proxmox-http/src/http/proxy_config.rs b/proxmox-http/src/http/proxy_config.rs
new file mode 100644
index 0000000..7e93a47
--- /dev/null
+++ b/proxmox-http/src/http/proxy_config.rs
@@ -0,0 +1,84 @@
+use anyhow::{Error, format_err, bail};
+
+use http::Uri;
+
+use crate::http::helpers;
+
+/// HTTP Proxy Configuration
+#[derive(Clone)]
+pub struct ProxyConfig {
+ pub host: String,
+ pub port: u16,
+ pub authorization: Option<String>, // user:pass
+ pub force_connect: bool,
+}
+
+impl ProxyConfig {
+ /// Parse proxy config from ALL_PROXY environment var
+ pub fn from_proxy_env() -> Result<Option<ProxyConfig>, Error> {
+
+ // We only support/use ALL_PROXY environment
+
+ match std::env::var_os("ALL_PROXY") {
+ None => return Ok(None),
+ Some(all_proxy) => {
+ let all_proxy = match all_proxy.to_str() {
+ Some(s) => String::from(s),
+ None => bail!("non UTF-8 content in env ALL_PROXY"),
+ };
+ if all_proxy.is_empty() {
+ return Ok(None);
+ }
+ let config = Self::parse_proxy_url(&all_proxy)?;
+ Ok(Some(config))
+ }
+ }
+ }
+
+ /// Parse proxy configuration string [http://]<host>[:port]
+ ///
+ /// Default port is 1080 (like curl)
+ pub fn parse_proxy_url(http_proxy: &str) -> Result<ProxyConfig, Error> {
+ proxmox::try_block!({
+ let proxy_uri: Uri = http_proxy.parse()?;
+ let proxy_authority = match proxy_uri.authority() {
+ Some(authority) => authority,
+ None => bail!("missing proxy authority"),
+ };
+ let host = proxy_authority.host().to_owned();
+ let port = match proxy_uri.port() {
+ Some(port) => port.as_u16(),
+ None => 1080, // CURL default port
+ };
+
+ match proxy_uri.scheme_str() {
+ Some("http") => { /* Ok */ }
+ Some(scheme) => bail!("unsupported proxy scheme '{}'", scheme),
+ None => { /* assume HTTP */ }
+ }
+
+ let authority_vec: Vec<&str> = proxy_authority.as_str().rsplitn(2, '@').collect();
+ let authorization = if authority_vec.len() == 2 {
+ Some(authority_vec[1].to_string())
+ } else {
+ None
+ };
+
+ Ok(ProxyConfig {
+ host,
+ port,
+ authorization,
+ force_connect: false,
+ })
+ }).map_err(|err| format_err!("parse_proxy_url failed: {}", err))
+ }
+
+ /// Assemble canonical proxy string (including scheme and port)
+ pub fn to_proxy_string(&self) -> Result<String, Error> {
+ let authority = helpers::build_authority(&self.host, self.port)?;
+ Ok(match self.authorization {
+ None => format!("http://{}", authority),
+ Some(ref authorization) => format!("http://{}@{}", authorization, authority)
+ })
+ }
+}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (7 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client " Fabian Grünbichler
` (11 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
the parts that were not already moved.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 2 +-
proxmox-http/src/http/client.rs | 3 +
proxmox-http/src/http/client/connector.rs | 203 ++++++++++++++++++++++
proxmox-http/src/http/mod.rs | 3 +
proxmox-http/src/lib.rs | 2 +-
5 files changed, 211 insertions(+), 2 deletions(-)
create mode 100644 proxmox-http/src/http/client.rs
create mode 100644 proxmox-http/src/http/client/connector.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 9c6fc35..cc3eef6 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -26,6 +26,6 @@ proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-fe
[features]
default = []
-client = [ "http-helpers" ]
+client = [ "futures", "http-helpers", "openssl" ]
http-helpers = [ "base64", "http", "hyper", "tokio/io-util", "tokio-openssl", "proxmox" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/client.rs b/proxmox-http/src/http/client.rs
new file mode 100644
index 0000000..21a65e3
--- /dev/null
+++ b/proxmox-http/src/http/client.rs
@@ -0,0 +1,3 @@
+mod connector;
+
+pub use connector::HttpsConnector;
diff --git a/proxmox-http/src/http/client/connector.rs b/proxmox-http/src/http/client/connector.rs
new file mode 100644
index 0000000..a302dd1
--- /dev/null
+++ b/proxmox-http/src/http/client/connector.rs
@@ -0,0 +1,203 @@
+use anyhow::{Error, format_err, bail};
+use std::os::unix::io::AsRawFd;
+use std::pin::Pin;
+use std::sync::Arc;
+use std::task::{Context, Poll};
+
+use futures::*;
+use http::Uri;
+use hyper::client::HttpConnector;
+use openssl::ssl::SslConnector;
+use tokio::io::{
+ AsyncRead,
+ AsyncReadExt,
+ AsyncWriteExt,
+};
+use tokio::net::TcpStream;
+use tokio_openssl::SslStream;
+
+use proxmox::sys::linux::socket::set_tcp_keepalive;
+
+use crate::http::{helpers, MaybeTlsStream, ProxyConfig};
+
+#[derive(Clone)]
+pub struct HttpsConnector {
+ connector: HttpConnector,
+ ssl_connector: Arc<SslConnector>,
+ proxy: Option<ProxyConfig>,
+ tcp_keepalive: u32,
+}
+
+impl HttpsConnector {
+ pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
+ connector.enforce_http(false);
+ Self {
+ connector,
+ ssl_connector: Arc::new(ssl_connector),
+ proxy: None,
+ tcp_keepalive,
+ }
+ }
+
+ pub fn set_proxy(&mut self, proxy: ProxyConfig) {
+ self.proxy = Some(proxy);
+ }
+
+ async fn secure_stream(
+ tcp_stream: TcpStream,
+ ssl_connector: &SslConnector,
+ host: &str,
+ ) -> Result<MaybeTlsStream<TcpStream>, Error> {
+ let config = ssl_connector.configure()?;
+ let mut conn: SslStream<TcpStream> = SslStream::new(config.into_ssl(host)?, tcp_stream)?;
+ Pin::new(&mut conn).connect().await?;
+ Ok(MaybeTlsStream::Secured(conn))
+ }
+
+ fn parse_status_line(status_line: &str) -> Result<(), Error> {
+ if !(status_line.starts_with("HTTP/1.1 200") || status_line.starts_with("HTTP/1.0 200")) {
+ bail!("proxy connect failed - invalid status: {}", status_line)
+ }
+ Ok(())
+ }
+
+ async fn parse_connect_response<R: AsyncRead + Unpin>(
+ stream: &mut R,
+ ) -> Result<(), Error> {
+
+ let mut data: Vec<u8> = Vec::new();
+ let mut buffer = [0u8; 256];
+ const END_MARK: &[u8; 4] = b"\r\n\r\n";
+
+ 'outer: loop {
+ let n = stream.read(&mut buffer[..]).await?;
+ if n == 0 { break; }
+ let search_start = if data.len() > END_MARK.len() { data.len() - END_MARK.len() + 1 } else { 0 };
+ data.extend(&buffer[..n]);
+ if data.len() >= END_MARK.len() {
+ if let Some(pos) = data[search_start..].windows(END_MARK.len()).position(|w| w == END_MARK) {
+ let response = String::from_utf8_lossy(&data);
+ let status_line = match response.split("\r\n").next() {
+ Some(status) => status,
+ None => bail!("missing newline"),
+ };
+ Self::parse_status_line(status_line)?;
+
+ if pos != data.len() - END_MARK.len() {
+ bail!("unexpected data after connect response");
+ }
+ break 'outer;
+ }
+ }
+ if data.len() > 1024*32 { // max 32K (random chosen limit)
+ bail!("too many bytes");
+ }
+ }
+ Ok(())
+ }
+}
+
+impl hyper::service::Service<Uri> for HttpsConnector {
+ type Response = MaybeTlsStream<TcpStream>;
+ type Error = Error;
+ #[allow(clippy::type_complexity)]
+ type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
+
+ fn poll_ready(&mut self, ctx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
+ self.connector
+ .poll_ready(ctx)
+ .map_err(|err| err.into())
+ }
+
+ fn call(&mut self, dst: Uri) -> Self::Future {
+ let mut connector = self.connector.clone();
+ let ssl_connector = Arc::clone(&self.ssl_connector);
+ let is_https = dst.scheme() == Some(&http::uri::Scheme::HTTPS);
+ let host = match dst.host() {
+ Some(host) => host.to_owned(),
+ None => {
+ return futures::future::err(format_err!("missing URL scheme")).boxed();
+ }
+ };
+ let port = dst.port_u16().unwrap_or(if is_https { 443 } else { 80 });
+ let keepalive = self.tcp_keepalive;
+
+ if let Some(ref proxy) = self.proxy {
+
+ let use_connect = is_https || proxy.force_connect;
+
+ let proxy_authority = match helpers::build_authority(&proxy.host, proxy.port) {
+ Ok(authority) => authority,
+ Err(err) => return futures::future::err(err).boxed(),
+ };
+
+ let proxy_uri = match Uri::builder()
+ .scheme("http")
+ .authority(proxy_authority.as_str())
+ .path_and_query("/")
+ .build()
+ {
+ Ok(uri) => uri,
+ Err(err) => return futures::future::err(err.into()).boxed(),
+ };
+
+ let authorization = proxy.authorization.clone();
+
+ if use_connect {
+ async move {
+
+ let mut tcp_stream = connector
+ .call(proxy_uri)
+ .await
+ .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+
+ let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
+ if let Some(authorization) = authorization {
+ connect_request.push_str(&format!("Proxy-Authorization: {}\r\n", authorization));
+ }
+ connect_request.push_str(&format!("Host: {0}:{1}\r\n\r\n", host, port));
+
+ tcp_stream.write_all(connect_request.as_bytes()).await?;
+ tcp_stream.flush().await?;
+
+ Self::parse_connect_response(&mut tcp_stream).await?;
+
+ if is_https {
+ Self::secure_stream(tcp_stream, &ssl_connector, &host).await
+ } else {
+ Ok(MaybeTlsStream::Normal(tcp_stream))
+ }
+ }.boxed()
+ } else {
+ async move {
+ let tcp_stream = connector
+ .call(proxy_uri)
+ .await
+ .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+
+ Ok(MaybeTlsStream::Proxied(tcp_stream))
+ }.boxed()
+ }
+ } else {
+ async move {
+ let dst_str = dst.to_string(); // for error messages
+ let tcp_stream = connector
+ .call(dst)
+ .await
+ .map_err(|err| format_err!("error connecting to {} - {}", dst_str, err))?;
+
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+
+ if is_https {
+ Self::secure_stream(tcp_stream, &ssl_connector, &host).await
+ } else {
+ Ok(MaybeTlsStream::Normal(tcp_stream))
+ }
+ }.boxed()
+ }
+ }
+}
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
index 055648e..3803713 100644
--- a/proxmox-http/src/http/mod.rs
+++ b/proxmox-http/src/http/mod.rs
@@ -5,3 +5,6 @@ pub mod helpers;
mod proxy_config;
pub use proxy_config::ProxyConfig;
+
+#[cfg(feature = "client")]
+pub mod client;
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
index 5e3f9ec..aa44c0d 100644
--- a/proxmox-http/src/lib.rs
+++ b/proxmox-http/src/lib.rs
@@ -1,5 +1,5 @@
#[cfg(feature = "websocket")]
pub mod websocket;
-#[cfg(feature = "http-helpers")]
+#[cfg(any(feature = "http-helpers", feature = "client"))]
pub mod http;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (8 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy Fabian Grünbichler
` (10 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
adapted to use already moved helpers/code.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/http/client.rs | 4 +
proxmox-http/src/http/client/simple.rs | 157 +++++++++++++++++++++++++
2 files changed, 161 insertions(+)
create mode 100644 proxmox-http/src/http/client/simple.rs
diff --git a/proxmox-http/src/http/client.rs b/proxmox-http/src/http/client.rs
index 21a65e3..c55cbd4 100644
--- a/proxmox-http/src/http/client.rs
+++ b/proxmox-http/src/http/client.rs
@@ -1,3 +1,7 @@
mod connector;
pub use connector::HttpsConnector;
+
+mod simple;
+pub use simple::SimpleHttp;
+pub use simple::SimpleHttpOptions;
diff --git a/proxmox-http/src/http/client/simple.rs b/proxmox-http/src/http/client/simple.rs
new file mode 100644
index 0000000..110fa55
--- /dev/null
+++ b/proxmox-http/src/http/client/simple.rs
@@ -0,0 +1,157 @@
+use anyhow::{Error, format_err, bail};
+use std::collections::HashMap;
+
+use hyper::Body;
+use hyper::client::{Client, HttpConnector};
+use http::{Request, Response, HeaderValue};
+use openssl::ssl::{SslConnector, SslMethod};
+use futures::*;
+
+use crate::http::{
+ ProxyConfig,
+ client::HttpsConnector,
+};
+
+/// Options for a SimpleHttp client.
+#[derive(Default)]
+pub struct SimpleHttpOptions {
+ /// Proxy configuration
+ pub proxy_config: Option<ProxyConfig>,
+ /// `User-Agent` header value, defaults to `proxmox-simple-http-client/0.1`
+ pub user_agent: Option<String>,
+ /// TCP keepalive time, defaults to 7200
+ pub tcp_keepalive: Option<u32>,
+}
+
+impl SimpleHttpOptions {
+ fn get_proxy_authorization(&self) -> Option<String> {
+ if let Some(ref proxy_config) = self.proxy_config {
+ if !proxy_config.force_connect {
+ return proxy_config.authorization.clone();
+ }
+ }
+
+ None
+ }
+}
+
+/// Asyncrounous HTTP client implementation
+pub struct SimpleHttp {
+ client: Client<HttpsConnector, Body>,
+ options: SimpleHttpOptions,
+}
+
+impl SimpleHttp {
+ pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-simple-http-client/0.1";
+
+ pub fn new() -> Self {
+ Self::with_options(SimpleHttpOptions::default())
+ }
+
+ pub fn with_options(options: SimpleHttpOptions) -> Self {
+ let ssl_connector = SslConnector::builder(SslMethod::tls()).unwrap().build();
+ Self::with_ssl_connector(ssl_connector, options)
+ }
+
+ pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
+ let connector = HttpConnector::new();
+ let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
+ if let Some(ref proxy_config) = options.proxy_config {
+ https.set_proxy(proxy_config.clone());
+ }
+ let client = Client::builder().build(https);
+ Self { client, options }
+ }
+
+ pub fn set_user_agent(&mut self, user_agent: &str) -> Result<(), Error> {
+ self.options.user_agent = Some(user_agent.to_owned());
+ Ok(())
+ }
+
+ fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
+ if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
+ if let Some(ref authorization) = self.options.get_proxy_authorization() {
+ request
+ .headers_mut()
+ .insert(
+ http::header::PROXY_AUTHORIZATION,
+ HeaderValue::from_str(authorization)?,
+ );
+ }
+ }
+ Ok(())
+ }
+
+ pub async fn request(&self, mut request: Request<Body>) -> Result<Response<Body>, Error> {
+ let user_agent = if let Some(ref user_agent) = self.options.user_agent {
+ HeaderValue::from_str(&user_agent)?
+ } else {
+ HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
+ };
+
+ request.headers_mut().insert(hyper::header::USER_AGENT, user_agent);
+
+ self.add_proxy_headers(&mut request)?;
+
+ self.client.request(request)
+ .map_err(Error::from)
+ .await
+ }
+
+ pub async fn post(
+ &mut self,
+ uri: &str,
+ body: Option<String>,
+ content_type: Option<&str>,
+ ) -> Result<Response<Body>, Error> {
+
+ let body = if let Some(body) = body {
+ Body::from(body)
+ } else {
+ Body::empty()
+ };
+ let content_type = content_type.unwrap_or("application/json");
+
+ let request = Request::builder()
+ .method("POST")
+ .uri(uri)
+ .header(hyper::header::CONTENT_TYPE, content_type)
+ .body(body)?;
+
+ self.request(request).await
+ }
+
+ pub async fn get_string(
+ &mut self,
+ uri: &str,
+ extra_headers: Option<&HashMap<String, String>>,
+ ) -> Result<String, Error> {
+
+ let mut request = Request::builder()
+ .method("GET")
+ .uri(uri);
+
+ if let Some(hs) = extra_headers {
+ for (h, v) in hs.iter() {
+ request = request.header(h, v);
+ }
+ }
+
+ let request = request.body(Body::empty())?;
+
+ let res = self.request(request).await?;
+
+ let status = res.status();
+ if !status.is_success() {
+ bail!("Got bad status '{}' from server", status)
+ }
+
+ Self::response_body_string(res).await
+ }
+
+ pub async fn response_body_string(res: Response<Body>) -> Result<String, Error> {
+ let buf = hyper::body::to_bytes(res).await?;
+ String::from_utf8(buf.to_vec())
+ .map_err(|err| format_err!("Error converting HTTP result data: {}", err))
+ }
+}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (9 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 12/13] http: rustfmt Fabian Grünbichler
` (9 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
again.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/http/client/simple.rs | 6 ++++++
proxmox-http/src/http/proxy_config.rs | 2 +-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/proxmox-http/src/http/client/simple.rs b/proxmox-http/src/http/client/simple.rs
index 110fa55..6783781 100644
--- a/proxmox-http/src/http/client/simple.rs
+++ b/proxmox-http/src/http/client/simple.rs
@@ -155,3 +155,9 @@ impl SimpleHttp {
.map_err(|err| format_err!("Error converting HTTP result data: {}", err))
}
}
+
+impl Default for SimpleHttp {
+ fn default() -> Self {
+ Self::new()
+ }
+}
diff --git a/proxmox-http/src/http/proxy_config.rs b/proxmox-http/src/http/proxy_config.rs
index 7e93a47..9eac3bd 100644
--- a/proxmox-http/src/http/proxy_config.rs
+++ b/proxmox-http/src/http/proxy_config.rs
@@ -20,7 +20,7 @@ impl ProxyConfig {
// We only support/use ALL_PROXY environment
match std::env::var_os("ALL_PROXY") {
- None => return Ok(None),
+ None => Ok(None),
Some(all_proxy) => {
let all_proxy = match all_proxy.to_str() {
Some(s) => String::from(s),
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 12/13] http: rustfmt
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (10 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 13/13] http: update d/control Fabian Grünbichler
` (8 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/http/client/connector.rs | 78 ++++++++++++-----------
proxmox-http/src/http/client/simple.rs | 45 ++++++-------
proxmox-http/src/http/helpers.rs | 11 ++--
proxmox-http/src/http/proxy_config.rs | 8 +--
proxmox-http/src/http/wrapper.rs | 65 +++++--------------
5 files changed, 89 insertions(+), 118 deletions(-)
diff --git a/proxmox-http/src/http/client/connector.rs b/proxmox-http/src/http/client/connector.rs
index a302dd1..d69fdf2 100644
--- a/proxmox-http/src/http/client/connector.rs
+++ b/proxmox-http/src/http/client/connector.rs
@@ -1,4 +1,4 @@
-use anyhow::{Error, format_err, bail};
+use anyhow::{bail, format_err, Error};
use std::os::unix::io::AsRawFd;
use std::pin::Pin;
use std::sync::Arc;
@@ -8,11 +8,7 @@ use futures::*;
use http::Uri;
use hyper::client::HttpConnector;
use openssl::ssl::SslConnector;
-use tokio::io::{
- AsyncRead,
- AsyncReadExt,
- AsyncWriteExt,
-};
+use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio_openssl::SslStream;
@@ -29,7 +25,11 @@ pub struct HttpsConnector {
}
impl HttpsConnector {
- pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
+ pub fn with_connector(
+ mut connector: HttpConnector,
+ ssl_connector: SslConnector,
+ tcp_keepalive: u32,
+ ) -> Self {
connector.enforce_http(false);
Self {
connector,
@@ -61,21 +61,27 @@ impl HttpsConnector {
Ok(())
}
- async fn parse_connect_response<R: AsyncRead + Unpin>(
- stream: &mut R,
- ) -> Result<(), Error> {
-
+ async fn parse_connect_response<R: AsyncRead + Unpin>(stream: &mut R) -> Result<(), Error> {
let mut data: Vec<u8> = Vec::new();
let mut buffer = [0u8; 256];
const END_MARK: &[u8; 4] = b"\r\n\r\n";
'outer: loop {
let n = stream.read(&mut buffer[..]).await?;
- if n == 0 { break; }
- let search_start = if data.len() > END_MARK.len() { data.len() - END_MARK.len() + 1 } else { 0 };
+ if n == 0 {
+ break;
+ }
+ let search_start = if data.len() > END_MARK.len() {
+ data.len() - END_MARK.len() + 1
+ } else {
+ 0
+ };
data.extend(&buffer[..n]);
if data.len() >= END_MARK.len() {
- if let Some(pos) = data[search_start..].windows(END_MARK.len()).position(|w| w == END_MARK) {
+ if let Some(pos) = data[search_start..]
+ .windows(END_MARK.len())
+ .position(|w| w == END_MARK)
+ {
let response = String::from_utf8_lossy(&data);
let status_line = match response.split("\r\n").next() {
Some(status) => status,
@@ -89,7 +95,8 @@ impl HttpsConnector {
break 'outer;
}
}
- if data.len() > 1024*32 { // max 32K (random chosen limit)
+ if data.len() > 1024 * 32 {
+ // max 32K (random chosen limit)
bail!("too many bytes");
}
}
@@ -101,12 +108,11 @@ impl hyper::service::Service<Uri> for HttpsConnector {
type Response = MaybeTlsStream<TcpStream>;
type Error = Error;
#[allow(clippy::type_complexity)]
- type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
+ type Future =
+ Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
fn poll_ready(&mut self, ctx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
- self.connector
- .poll_ready(ctx)
- .map_err(|err| err.into())
+ self.connector.poll_ready(ctx).map_err(|err| err.into())
}
fn call(&mut self, dst: Uri) -> Self::Future {
@@ -123,7 +129,6 @@ impl hyper::service::Service<Uri> for HttpsConnector {
let keepalive = self.tcp_keepalive;
if let Some(ref proxy) = self.proxy {
-
let use_connect = is_https || proxy.force_connect;
let proxy_authority = match helpers::build_authority(&proxy.host, proxy.port) {
@@ -145,17 +150,16 @@ impl hyper::service::Service<Uri> for HttpsConnector {
if use_connect {
async move {
-
- let mut tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+ let mut tcp_stream = connector.call(proxy_uri).await.map_err(|err| {
+ format_err!("error connecting to {} - {}", proxy_authority, err)
+ })?;
let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
if let Some(authorization) = authorization {
- connect_request.push_str(&format!("Proxy-Authorization: {}\r\n", authorization));
+ connect_request
+ .push_str(&format!("Proxy-Authorization: {}\r\n", authorization));
}
connect_request.push_str(&format!("Host: {0}:{1}\r\n\r\n", host, port));
@@ -169,18 +173,19 @@ impl hyper::service::Service<Uri> for HttpsConnector {
} else {
Ok(MaybeTlsStream::Normal(tcp_stream))
}
- }.boxed()
+ }
+ .boxed()
} else {
- async move {
- let tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+ async move {
+ let tcp_stream = connector.call(proxy_uri).await.map_err(|err| {
+ format_err!("error connecting to {} - {}", proxy_authority, err)
+ })?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
- Ok(MaybeTlsStream::Proxied(tcp_stream))
- }.boxed()
+ Ok(MaybeTlsStream::Proxied(tcp_stream))
+ }
+ .boxed()
}
} else {
async move {
@@ -197,7 +202,8 @@ impl hyper::service::Service<Uri> for HttpsConnector {
} else {
Ok(MaybeTlsStream::Normal(tcp_stream))
}
- }.boxed()
+ }
+ .boxed()
}
}
}
diff --git a/proxmox-http/src/http/client/simple.rs b/proxmox-http/src/http/client/simple.rs
index 6783781..b6e6d5a 100644
--- a/proxmox-http/src/http/client/simple.rs
+++ b/proxmox-http/src/http/client/simple.rs
@@ -1,16 +1,13 @@
-use anyhow::{Error, format_err, bail};
+use anyhow::{bail, format_err, Error};
use std::collections::HashMap;
-use hyper::Body;
+use futures::*;
+use http::{HeaderValue, Request, Response};
use hyper::client::{Client, HttpConnector};
-use http::{Request, Response, HeaderValue};
+use hyper::Body;
use openssl::ssl::{SslConnector, SslMethod};
-use futures::*;
-use crate::http::{
- ProxyConfig,
- client::HttpsConnector,
-};
+use crate::http::{client::HttpsConnector, ProxyConfig};
/// Options for a SimpleHttp client.
#[derive(Default)]
@@ -27,7 +24,7 @@ impl SimpleHttpOptions {
fn get_proxy_authorization(&self) -> Option<String> {
if let Some(ref proxy_config) = self.proxy_config {
if !proxy_config.force_connect {
- return proxy_config.authorization.clone();
+ return proxy_config.authorization.clone();
}
}
@@ -55,7 +52,11 @@ impl SimpleHttp {
pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
+ let mut https = HttpsConnector::with_connector(
+ connector,
+ ssl_connector,
+ options.tcp_keepalive.unwrap_or(7200),
+ );
if let Some(ref proxy_config) = options.proxy_config {
https.set_proxy(proxy_config.clone());
}
@@ -71,12 +72,10 @@ impl SimpleHttp {
fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
if let Some(ref authorization) = self.options.get_proxy_authorization() {
- request
- .headers_mut()
- .insert(
- http::header::PROXY_AUTHORIZATION,
- HeaderValue::from_str(authorization)?,
- );
+ request.headers_mut().insert(
+ http::header::PROXY_AUTHORIZATION,
+ HeaderValue::from_str(authorization)?,
+ );
}
}
Ok(())
@@ -89,13 +88,13 @@ impl SimpleHttp {
HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
};
- request.headers_mut().insert(hyper::header::USER_AGENT, user_agent);
+ request
+ .headers_mut()
+ .insert(hyper::header::USER_AGENT, user_agent);
self.add_proxy_headers(&mut request)?;
- self.client.request(request)
- .map_err(Error::from)
- .await
+ self.client.request(request).map_err(Error::from).await
}
pub async fn post(
@@ -104,7 +103,6 @@ impl SimpleHttp {
body: Option<String>,
content_type: Option<&str>,
) -> Result<Response<Body>, Error> {
-
let body = if let Some(body) = body {
Body::from(body)
} else {
@@ -126,10 +124,7 @@ impl SimpleHttp {
uri: &str,
extra_headers: Option<&HashMap<String, String>>,
) -> Result<String, Error> {
-
- let mut request = Request::builder()
- .method("GET")
- .uri(uri);
+ let mut request = Request::builder().method("GET").uri(uri);
if let Some(hs) = extra_headers {
for (h, v) in hs.iter() {
diff --git a/proxmox-http/src/http/helpers.rs b/proxmox-http/src/http/helpers.rs
index 3f663d2..ca67161 100644
--- a/proxmox-http/src/http/helpers.rs
+++ b/proxmox-http/src/http/helpers.rs
@@ -6,10 +6,11 @@ use http::uri::Authority;
pub fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
let bytes = host.as_bytes();
let len = bytes.len();
- let authority = if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len-1] != b']' {
- format!("[{}]:{}", host, port).parse()?
- } else {
- format!("{}:{}", host, port).parse()?
- };
+ let authority =
+ if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len - 1] != b']' {
+ format!("[{}]:{}", host, port).parse()?
+ } else {
+ format!("{}:{}", host, port).parse()?
+ };
Ok(authority)
}
diff --git a/proxmox-http/src/http/proxy_config.rs b/proxmox-http/src/http/proxy_config.rs
index 9eac3bd..3fe7cba 100644
--- a/proxmox-http/src/http/proxy_config.rs
+++ b/proxmox-http/src/http/proxy_config.rs
@@ -1,4 +1,4 @@
-use anyhow::{Error, format_err, bail};
+use anyhow::{bail, format_err, Error};
use http::Uri;
@@ -16,7 +16,6 @@ pub struct ProxyConfig {
impl ProxyConfig {
/// Parse proxy config from ALL_PROXY environment var
pub fn from_proxy_env() -> Result<Option<ProxyConfig>, Error> {
-
// We only support/use ALL_PROXY environment
match std::env::var_os("ALL_PROXY") {
@@ -70,7 +69,8 @@ impl ProxyConfig {
authorization,
force_connect: false,
})
- }).map_err(|err| format_err!("parse_proxy_url failed: {}", err))
+ })
+ .map_err(|err| format_err!("parse_proxy_url failed: {}", err))
}
/// Assemble canonical proxy string (including scheme and port)
@@ -78,7 +78,7 @@ impl ProxyConfig {
let authority = helpers::build_authority(&self.host, self.port)?;
Ok(match self.authorization {
None => format!("http://{}", authority),
- Some(ref authorization) => format!("http://{}@{}", authorization, authority)
+ Some(ref authorization) => format!("http://{}@{}", authorization, authority),
})
}
}
diff --git a/proxmox-http/src/http/wrapper.rs b/proxmox-http/src/http/wrapper.rs
index 3399b28..9ed4221 100644
--- a/proxmox-http/src/http/wrapper.rs
+++ b/proxmox-http/src/http/wrapper.rs
@@ -2,7 +2,7 @@ use std::io;
use std::pin::Pin;
use std::task::{Context, Poll};
-use hyper::client::connect::{Connection, Connected};
+use hyper::client::connect::{Connected, Connection};
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
use tokio_openssl::SslStream;
@@ -22,15 +22,9 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
buf: &mut ReadBuf,
) -> Poll<Result<(), io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_read(cx, buf),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_read(cx, buf),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_read(cx, buf),
}
}
}
@@ -42,15 +36,9 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
buf: &[u8],
) -> Poll<Result<usize, io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_write(cx, buf),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_write(cx, buf),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_write(cx, buf),
}
}
@@ -60,15 +48,9 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
bufs: &[io::IoSlice<'_>],
) -> Poll<Result<usize, io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_write_vectored(cx, bufs),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_write_vectored(cx, bufs),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_write_vectored(cx, bufs),
}
}
@@ -82,36 +64,23 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_flush(cx),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_flush(cx),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_flush(cx),
}
}
fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_shutdown(cx),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_shutdown(cx),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_shutdown(cx),
}
}
}
// we need this for the hyper http client
-impl <S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S>
-{
+impl<S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S> {
fn connected(&self) -> Connected {
match self {
MaybeTlsStream::Normal(s) => s.connected(),
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 13/13] http: update d/control
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (11 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 12/13] http: rustfmt Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
` (7 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/debian/control | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/proxmox-http/debian/control b/proxmox-http/debian/control
index f010d6e..860e705 100644
--- a/proxmox-http/debian/control
+++ b/proxmox-http/debian/control
@@ -118,18 +118,12 @@ Depends:
librust-tokio-1+io-util-dev,
librust-tokio-openssl-0.6+default-dev (>= 0.6.1-~~)
Provides:
- librust-proxmox-http+simple-client-dev (= ${binary:Version}),
librust-proxmox-http-0+http-helpers-dev (= ${binary:Version}),
- librust-proxmox-http-0+simple-client-dev (= ${binary:Version}),
librust-proxmox-http-0.1+http-helpers-dev (= ${binary:Version}),
- librust-proxmox-http-0.1+simple-client-dev (= ${binary:Version}),
- librust-proxmox-http-0.1.0+http-helpers-dev (= ${binary:Version}),
- librust-proxmox-http-0.1.0+simple-client-dev (= ${binary:Version})
-Description: Proxmox HTTP library - feature "http-helpers" and 1 more
+ librust-proxmox-http-0.1.0+http-helpers-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "http-helpers"
This metapackage enables feature "http-helpers" for the Rust proxmox-http
crate, by pulling in any additional dependencies needed by that feature.
- .
- Additionally, this package also provides the "simple-client" feature.
Package: librust-proxmox-http+hyper-dev
Architecture: any
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (12 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 13/13] http: update d/control Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-17 8:35 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate Fabian Grünbichler
` (6 subsequent siblings)
20 siblings, 1 reply; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires patch #3 in proxmox
Cargo.toml | 5 +++--
debian/control | 3 ++-
src/api2/node.rs | 2 +-
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index cd99241e..997017c0 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -53,10 +53,11 @@ percent-encoding = "2.1"
pin-utils = "0.1.0"
pin-project = "1.0"
pathpatterns = "0.1.2"
-proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro", "websocket" ] }
+proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
-#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro", "websocket" ] }
+#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
proxmox-fuse = "0.1.1"
+proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "websocket" ] }
pxar = { version = "0.10.1", features = [ "tokio-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io" ] }
regex = "1.2"
diff --git a/debian/control b/debian/control
index 3f1edba8..a02518fe 100644
--- a/debian/control
+++ b/debian/control
@@ -42,9 +42,10 @@ Build-Depends: debhelper (>= 11),
librust-proxmox-0.11+api-macro-dev (>= 0.11.4-~~),
librust-proxmox-0.11+default-dev (>= 0.11.4-~~),
librust-proxmox-0.11+sortable-macro-dev (>= 0.11.4-~~),
- librust-proxmox-0.11+websocket-dev (>= 0.11.4-~~),
librust-proxmox-acme-rs-0.2+default-dev (>= 0.2.1-~~),
librust-proxmox-fuse-0.1+default-dev (>= 0.1.1-~~),
+ librust-proxmox-http-0.1+default-dev,
+ librust-proxmox-http-0.1+websocket-dev,
librust-pxar-0.10+default-dev (>= 0.10.1-~~),
librust-pxar-0.10+tokio-io-dev (>= 0.10.1-~~),
librust-regex-1+default-dev (>= 1.2-~~),
diff --git a/src/api2/node.rs b/src/api2/node.rs
index 75271cd5..af9ecf69 100644
--- a/src/api2/node.rs
+++ b/src/api2/node.rs
@@ -17,7 +17,7 @@ use proxmox::api::{
api, schema::*, ApiHandler, ApiMethod, ApiResponseFuture, Permission, RpcEnvironment,
};
use proxmox::list_subdirs_api_method;
-use proxmox::tools::websocket::WebSocket;
+use proxmox_http::websocket::WebSocket;
use proxmox::{identity, sortable};
use crate::api2::types::*;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (13 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http Fabian Grünbichler
` (5 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
and constant to tools module.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #5 and proxmox version bump
src/bin/proxmox-backup-proxy.rs | 6 ++----
src/tools.rs | 4 +++-
src/tools/http.rs | 7 +++----
src/tools/socket.rs | 23 -----------------------
4 files changed, 8 insertions(+), 32 deletions(-)
delete mode 100644 src/tools/socket.rs
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 71517023..a53f554a 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -11,6 +11,7 @@ use serde_json::Value;
use proxmox::try_block;
use proxmox::api::RpcEnvironmentType;
+use proxmox::sys::linux::socket::set_tcp_keepalive;
use proxmox_backup::{
backup::DataStore,
@@ -38,6 +39,7 @@ use proxmox_backup::buildcfg;
use proxmox_backup::server;
use proxmox_backup::auth_helpers::*;
use proxmox_backup::tools::{
+ PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
daemon,
disks::{
DiskManage,
@@ -45,10 +47,6 @@ use proxmox_backup::tools::{
get_pool_from_dataset,
},
logrotate::LogRotate,
- socket::{
- set_tcp_keepalive,
- PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- },
};
use proxmox_backup::api2::pull::do_sync_job;
diff --git a/src/tools.rs b/src/tools.rs
index 8f2acded..8a1d0bc7 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -43,7 +43,6 @@ pub mod lru_cache;
pub mod nom;
pub mod runtime;
pub mod serde_filter;
-pub mod socket;
pub mod statistics;
pub mod subscription;
pub mod systemd;
@@ -483,6 +482,9 @@ impl<T: Any> AsAny for T {
}
}
+/// The default 2 hours are far too long for PBS
+pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
+
/// This used to be: `SIMPLE_ENCODE_SET` plus space, `"`, `#`, `<`, `>`, backtick, `?`, `{`, `}`
pub const DEFAULT_ENCODE_SET: &AsciiSet = &percent_encoding::CONTROLS // 0..1f and 7e
// The SIMPLE_ENCODE_SET adds space and anything >= 0x7e (7e itself is already included above)
diff --git a/src/tools/http.rs b/src/tools/http.rs
index 0821992a..1d96c70f 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -18,12 +18,11 @@ use tokio::{
};
use tokio_openssl::SslStream;
+use proxmox::sys::linux::socket::set_tcp_keepalive;
+
use crate::tools::{
+ PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
async_io::MaybeTlsStream,
- socket::{
- set_tcp_keepalive,
- PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- },
};
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
diff --git a/src/tools/socket.rs b/src/tools/socket.rs
deleted file mode 100644
index 01e5edec..00000000
--- a/src/tools/socket.rs
+++ /dev/null
@@ -1,23 +0,0 @@
-use std::os::unix::io::RawFd;
-
-use nix::sys::socket::sockopt::{KeepAlive, TcpKeepIdle};
-use nix::sys::socket::setsockopt;
-
-pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
-
-/// Set TCP keepalive time on a socket
-///
-/// See "man 7 tcp" for details.
-///
-/// The default on Linux is 7200 (2 hours) which is far too long for
-/// our backup tools.
-pub fn set_tcp_keepalive(
- socket_fd: RawFd,
- tcp_keepalive_time: u32,
-) -> nix::Result<()> {
-
- setsockopt(socket_fd, KeepAlive, &true)?;
- setsockopt(socket_fd, TcpKeepIdle, &tcp_keepalive_time)?;
-
- Ok(())
-}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (14 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable Fabian Grünbichler
` (4 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #6
Cargo.toml | 2 +-
src/tools/async_io.rs | 119 +-----------------------------------------
src/tools/http.rs | 6 +--
3 files changed, 4 insertions(+), 123 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index 997017c0..df649c1b 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -57,7 +57,7 @@ proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
proxmox-fuse = "0.1.1"
-proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "websocket" ] }
+proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "http-helpers", "websocket" ] }
pxar = { version = "0.10.1", features = [ "tokio-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io" ] }
regex = "1.2"
diff --git a/src/tools/async_io.rs b/src/tools/async_io.rs
index 83110912..66d38094 100644
--- a/src/tools/async_io.rs
+++ b/src/tools/async_io.rs
@@ -1,131 +1,14 @@
//! AsyncRead/AsyncWrite utilities.
-use std::io;
use std::os::unix::io::{AsRawFd, RawFd};
use std::pin::Pin;
use std::task::{Context, Poll};
use futures::stream::{Stream, TryStream};
use futures::ready;
-use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
+use tokio::io::{AsyncRead, AsyncWrite};
use tokio::net::TcpListener;
-use tokio_openssl::SslStream;
-use hyper::client::connect::{Connection, Connected};
-/// Asynchronous stream, possibly encrypted and proxied
-///
-/// Usefule for HTTP client implementations using hyper.
-pub enum MaybeTlsStream<S> {
- Normal(S),
- Proxied(S),
- Secured(SslStream<S>),
-}
-
-impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
- fn poll_read(
- self: Pin<&mut Self>,
- cx: &mut Context,
- buf: &mut ReadBuf,
- ) -> Poll<Result<(), io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- }
- }
-}
-
-impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
- fn poll_write(
- self: Pin<&mut Self>,
- cx: &mut Context,
- buf: &[u8],
- ) -> Poll<Result<usize, io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- }
- }
-
- fn poll_write_vectored(
- self: Pin<&mut Self>,
- cx: &mut Context<'_>,
- bufs: &[io::IoSlice<'_>],
- ) -> Poll<Result<usize, io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- }
- }
-
- fn is_write_vectored(&self) -> bool {
- match self {
- MaybeTlsStream::Normal(s) => s.is_write_vectored(),
- MaybeTlsStream::Proxied(s) => s.is_write_vectored(),
- MaybeTlsStream::Secured(s) => s.is_write_vectored(),
- }
- }
-
- fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- }
- }
-
- fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- }
- }
-}
-
-// we need this for the hyper http client
-impl <S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S>
-{
- fn connected(&self) -> Connected {
- match self {
- MaybeTlsStream::Normal(s) => s.connected(),
- MaybeTlsStream::Proxied(s) => s.connected().proxy(true),
- MaybeTlsStream::Secured(s) => s.get_ref().connected(),
- }
- }
-}
/// Tokio's `Incoming` now is a reference type and hyper's `AddrIncoming` misses some standard
/// stuff like `AsRawFd`, so here's something implementing hyper's `Accept` from a `TcpListener`
diff --git a/src/tools/http.rs b/src/tools/http.rs
index 1d96c70f..0f5b8470 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -19,11 +19,9 @@ use tokio::{
use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
+use proxmox_http::http::MaybeTlsStream;
-use crate::tools::{
- PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- async_io::MaybeTlsStream,
-};
+use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (15 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http Fabian Grünbichler
` (3 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
it's the only PBS-specific part in there, so let's make it
product-agnostic before moving it off to proxmox-http.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
src/client/http_client.rs | 3 ++-
src/tools/http.rs | 13 +++++++------
src/tools/simple_http_client.rs | 3 ++-
3 files changed, 11 insertions(+), 8 deletions(-)
diff --git a/src/client/http_client.rs b/src/client/http_client.rs
index 7fe33bcc..056f30e5 100644
--- a/src/client/http_client.rs
+++ b/src/client/http_client.rs
@@ -26,6 +26,7 @@ use crate::tools::{
self,
BroadcastFuture,
DEFAULT_ENCODE_SET,
+ PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
http::{
build_authority,
HttpsConnector,
@@ -343,7 +344,7 @@ impl HttpClient {
httpc.enforce_http(false); // we want https...
httpc.set_connect_timeout(Some(std::time::Duration::new(10, 0)));
- let https = HttpsConnector::with_connector(httpc, ssl_connector_builder.build());
+ let https = HttpsConnector::with_connector(httpc, ssl_connector_builder.build(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
let client = Client::builder()
//.http2_initial_stream_window_size( (1 << 31) - 2)
diff --git a/src/tools/http.rs b/src/tools/http.rs
index 0f5b8470..a6b92aad 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -21,8 +21,6 @@ use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
use proxmox_http::http::MaybeTlsStream;
-use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
-
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
let bytes = host.as_bytes();
@@ -120,15 +118,17 @@ pub struct HttpsConnector {
connector: HttpConnector,
ssl_connector: Arc<SslConnector>,
proxy: Option<ProxyConfig>,
+ tcp_keepalive: u32,
}
impl HttpsConnector {
- pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector) -> Self {
+ pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
connector.enforce_http(false);
Self {
connector,
ssl_connector: Arc::new(ssl_connector),
proxy: None,
+ tcp_keepalive,
}
}
@@ -213,6 +213,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
}
};
let port = dst.port_u16().unwrap_or(if is_https { 443 } else { 80 });
+ let keepalive = self.tcp_keepalive;
if let Some(ref proxy) = self.proxy {
@@ -243,7 +244,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
.await
.map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
if let Some(authorization) = authorization {
@@ -272,7 +273,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
.await
.map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
Ok(MaybeTlsStream::Proxied(tcp_stream))
}.boxed()
@@ -285,7 +286,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
.await
.map_err(|err| format_err!("error connecting to {} - {}", dst_str, err))?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
if is_https {
Self::secure_stream(tcp_stream, &ssl_connector, &host).await
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index ca11ded8..1e399267 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -7,6 +7,7 @@ use http::{Request, Response, HeaderValue};
use openssl::ssl::{SslConnector, SslMethod};
use futures::*;
+use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
use crate::tools::http::{HttpsConnector, ProxyConfig};
/// Asyncrounous HTTP client implementation
@@ -35,7 +36,7 @@ impl SimpleHttp {
}
let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector);
+ let mut https = HttpsConnector::with_connector(connector, ssl_connector, PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
if let Some(proxy_config) = proxy_config {
https.set_proxy(proxy_config);
}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (16 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http " Fabian Grünbichler
` (2 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #7 & #8
src/api2/node/apt.rs | 4 +-
src/api2/types/mod.rs | 2 +-
src/config/node.rs | 3 +-
src/tools/http.rs | 82 +--------------------------------
src/tools/simple_http_client.rs | 4 +-
5 files changed, 10 insertions(+), 85 deletions(-)
diff --git a/src/api2/node/apt.rs b/src/api2/node/apt.rs
index c149a14b..120d5339 100644
--- a/src/api2/node/apt.rs
+++ b/src/api2/node/apt.rs
@@ -7,9 +7,11 @@ use proxmox::api::{api, RpcEnvironment, RpcEnvironmentType, Permission};
use proxmox::api::router::{Router, SubdirMap};
use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_http::http::ProxyConfig;
+
use crate::config::node;
use crate::server::WorkerTask;
-use crate::tools::{apt, SimpleHttp, http::ProxyConfig, subscription};
+use crate::tools::{apt, SimpleHttp, subscription};
use crate::config::acl::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY};
use crate::api2::types::{Authid, APTUpdateInfo, NODE_SCHEMA, UPID_SCHEMA};
diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index e42083f0..26b99790 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -1660,7 +1660,7 @@ pub struct NodeStatus {
pub const HTTP_PROXY_SCHEMA: Schema = StringSchema::new(
"HTTP proxy configuration [http://]<host>[:port]")
.format(&ApiStringFormat::VerifyFn(|s| {
- crate::tools::http::ProxyConfig::parse_proxy_url(s)?;
+ proxmox_http::http::ProxyConfig::parse_proxy_url(s)?;
Ok(())
}))
.min_length(1)
diff --git a/src/config/node.rs b/src/config/node.rs
index e818e47e..b003ae01 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -10,11 +10,12 @@ use proxmox::api::api;
use proxmox::api::schema::{ApiStringFormat, Updater};
use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_http::http::ProxyConfig;
+
use crate::acme::AcmeClient;
use crate::api2::types::{
AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
};
-use crate::tools::http::ProxyConfig;
const CONF_FILE: &str = configdir!("/node.cfg");
const LOCK_FILE: &str = configdir!("/.node.lck");
diff --git a/src/tools/http.rs b/src/tools/http.rs
index a6b92aad..b99d26a1 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -19,7 +19,7 @@ use tokio::{
use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
-use proxmox_http::http::MaybeTlsStream;
+use proxmox_http::http::{MaybeTlsStream, ProxyConfig};
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
@@ -33,86 +33,6 @@ pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error>
Ok(authority)
}
-/// HTTP Proxy Configuration
-#[derive(Clone)]
-pub struct ProxyConfig {
- pub host: String,
- pub port: u16,
- pub authorization: Option<String>, // user:pass
- pub force_connect: bool,
-}
-
-impl ProxyConfig {
-
- /// Parse proxy config from ALL_PROXY environment var
- pub fn from_proxy_env() -> Result<Option<ProxyConfig>, Error> {
-
- // We only support/use ALL_PROXY environment
-
- match std::env::var_os("ALL_PROXY") {
- None => return Ok(None),
- Some(all_proxy) => {
- let all_proxy = match all_proxy.to_str() {
- Some(s) => String::from(s),
- None => bail!("non UTF-8 content in env ALL_PROXY"),
- };
- if all_proxy.is_empty() {
- return Ok(None);
- }
- let config = Self::parse_proxy_url(&all_proxy)?;
- Ok(Some(config))
- }
- }
- }
-
- /// Parse proxy configuration string [http://]<host>[:port]
- ///
- /// Default port is 1080 (like curl)
- pub fn parse_proxy_url(http_proxy: &str) -> Result<ProxyConfig, Error> {
- proxmox::try_block!({
- let proxy_uri: Uri = http_proxy.parse()?;
- let proxy_authority = match proxy_uri.authority() {
- Some(authority) => authority,
- None => bail!("missing proxy authority"),
- };
- let host = proxy_authority.host().to_owned();
- let port = match proxy_uri.port() {
- Some(port) => port.as_u16(),
- None => 1080, // CURL default port
- };
-
- match proxy_uri.scheme_str() {
- Some("http") => { /* Ok */ }
- Some(scheme) => bail!("unsupported proxy scheme '{}'", scheme),
- None => { /* assume HTTP */ }
- }
-
- let authority_vec: Vec<&str> = proxy_authority.as_str().rsplitn(2, '@').collect();
- let authorization = if authority_vec.len() == 2 {
- Some(authority_vec[1].to_string())
- } else {
- None
- };
-
- Ok(ProxyConfig {
- host,
- port,
- authorization,
- force_connect: false,
- })
- }).map_err(|err| format_err!("parse_proxy_url failed: {}", err))
- }
-
- /// Assemble canonical proxy string (including scheme and port)
- pub fn to_proxy_string(&self) -> Result<String, Error> {
- let authority = build_authority(&self.host, self.port)?;
- Ok(match self.authorization {
- None => format!("http://{}", authority),
- Some(ref authorization) => format!("http://{}@{}", authorization, authority)
- })
- }
-}
-
#[derive(Clone)]
pub struct HttpsConnector {
connector: HttpConnector,
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index 1e399267..729711c8 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -7,8 +7,10 @@ use http::{Request, Response, HeaderValue};
use openssl::ssl::{SslConnector, SslMethod};
use futures::*;
+use proxmox_http::http::ProxyConfig;
+
use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
-use crate::tools::http::{HttpsConnector, ProxyConfig};
+use crate::tools::http::HttpsConnector;
/// Asyncrounous HTTP client implementation
pub struct SimpleHttp {
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (17 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http Fabian Grünbichler
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #7-9
Cargo.toml | 2 +-
src/client/http_client.rs | 7 +-
src/tools.rs | 1 -
src/tools/http.rs | 197 --------------------------------
src/tools/simple_http_client.rs | 6 +-
5 files changed, 8 insertions(+), 205 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index df649c1b..73bd4240 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -57,7 +57,7 @@ proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
proxmox-fuse = "0.1.1"
-proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "http-helpers", "websocket" ] }
+proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "client", "http-helpers", "websocket" ] }
pxar = { version = "0.10.1", features = [ "tokio-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io" ] }
regex = "1.2"
diff --git a/src/client/http_client.rs b/src/client/http_client.rs
index 056f30e5..4c548481 100644
--- a/src/client/http_client.rs
+++ b/src/client/http_client.rs
@@ -20,6 +20,9 @@ use proxmox::{
tools::fs::{file_get_json, replace_file, CreateOptions},
};
+use proxmox_http::http::client::HttpsConnector;
+use proxmox_http::http::helpers::build_authority;
+
use super::pipe_to_stream::PipeToSendStream;
use crate::api2::types::{Authid, Userid};
use crate::tools::{
@@ -27,10 +30,6 @@ use crate::tools::{
BroadcastFuture,
DEFAULT_ENCODE_SET,
PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- http::{
- build_authority,
- HttpsConnector,
- },
};
/// Timeout used for several HTTP operations that are expected to finish quickly but may block in
diff --git a/src/tools.rs b/src/tools.rs
index 8a1d0bc7..eb9a9fd0 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -31,7 +31,6 @@ pub mod disks;
pub mod format;
pub mod fs;
pub mod fuse_loop;
-pub mod http;
mod simple_http_client;
pub use simple_http_client::SimpleHttp;
diff --git a/src/tools/http.rs b/src/tools/http.rs
index b99d26a1..dba8dc98 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -20,200 +20,3 @@ use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
use proxmox_http::http::{MaybeTlsStream, ProxyConfig};
-
-// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
-pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
- let bytes = host.as_bytes();
- let len = bytes.len();
- let authority = if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len-1] != b']' {
- format!("[{}]:{}", host, port).parse()?
- } else {
- format!("{}:{}", host, port).parse()?
- };
- Ok(authority)
-}
-
-#[derive(Clone)]
-pub struct HttpsConnector {
- connector: HttpConnector,
- ssl_connector: Arc<SslConnector>,
- proxy: Option<ProxyConfig>,
- tcp_keepalive: u32,
-}
-
-impl HttpsConnector {
- pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
- connector.enforce_http(false);
- Self {
- connector,
- ssl_connector: Arc::new(ssl_connector),
- proxy: None,
- tcp_keepalive,
- }
- }
-
- pub fn set_proxy(&mut self, proxy: ProxyConfig) {
- self.proxy = Some(proxy);
- }
-
- async fn secure_stream(
- tcp_stream: TcpStream,
- ssl_connector: &SslConnector,
- host: &str,
- ) -> Result<MaybeTlsStream<TcpStream>, Error> {
- let config = ssl_connector.configure()?;
- let mut conn: SslStream<TcpStream> = SslStream::new(config.into_ssl(host)?, tcp_stream)?;
- Pin::new(&mut conn).connect().await?;
- Ok(MaybeTlsStream::Secured(conn))
- }
-
- fn parse_status_line(status_line: &str) -> Result<(), Error> {
- if !(status_line.starts_with("HTTP/1.1 200") || status_line.starts_with("HTTP/1.0 200")) {
- bail!("proxy connect failed - invalid status: {}", status_line)
- }
- Ok(())
- }
-
- async fn parse_connect_response<R: AsyncRead + Unpin>(
- stream: &mut R,
- ) -> Result<(), Error> {
-
- let mut data: Vec<u8> = Vec::new();
- let mut buffer = [0u8; 256];
- const END_MARK: &[u8; 4] = b"\r\n\r\n";
-
- 'outer: loop {
- let n = stream.read(&mut buffer[..]).await?;
- if n == 0 { break; }
- let search_start = if data.len() > END_MARK.len() { data.len() - END_MARK.len() + 1 } else { 0 };
- data.extend(&buffer[..n]);
- if data.len() >= END_MARK.len() {
- if let Some(pos) = data[search_start..].windows(END_MARK.len()).position(|w| w == END_MARK) {
- let response = String::from_utf8_lossy(&data);
- let status_line = match response.split("\r\n").next() {
- Some(status) => status,
- None => bail!("missing newline"),
- };
- Self::parse_status_line(status_line)?;
-
- if pos != data.len() - END_MARK.len() {
- bail!("unexpected data after connect response");
- }
- break 'outer;
- }
- }
- if data.len() > 1024*32 { // max 32K (random chosen limit)
- bail!("too many bytes");
- }
- }
- Ok(())
- }
-}
-
-impl hyper::service::Service<Uri> for HttpsConnector {
- type Response = MaybeTlsStream<TcpStream>;
- type Error = Error;
- #[allow(clippy::type_complexity)]
- type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
-
- fn poll_ready(&mut self, ctx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
- self.connector
- .poll_ready(ctx)
- .map_err(|err| err.into())
- }
-
- fn call(&mut self, dst: Uri) -> Self::Future {
- let mut connector = self.connector.clone();
- let ssl_connector = Arc::clone(&self.ssl_connector);
- let is_https = dst.scheme() == Some(&http::uri::Scheme::HTTPS);
- let host = match dst.host() {
- Some(host) => host.to_owned(),
- None => {
- return futures::future::err(format_err!("missing URL scheme")).boxed();
- }
- };
- let port = dst.port_u16().unwrap_or(if is_https { 443 } else { 80 });
- let keepalive = self.tcp_keepalive;
-
- if let Some(ref proxy) = self.proxy {
-
- let use_connect = is_https || proxy.force_connect;
-
- let proxy_authority = match build_authority(&proxy.host, proxy.port) {
- Ok(authority) => authority,
- Err(err) => return futures::future::err(err).boxed(),
- };
-
- let proxy_uri = match Uri::builder()
- .scheme("http")
- .authority(proxy_authority.as_str())
- .path_and_query("/")
- .build()
- {
- Ok(uri) => uri,
- Err(err) => return futures::future::err(err.into()).boxed(),
- };
-
- let authorization = proxy.authorization.clone();
-
- if use_connect {
- async move {
-
- let mut tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
-
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
-
- let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
- if let Some(authorization) = authorization {
- connect_request.push_str(&format!(
- "Proxy-Authorization: Basic {}\r\n",
- base64::encode(authorization),
- ));
- }
- connect_request.push_str(&format!("Host: {0}:{1}\r\n\r\n", host, port));
-
- tcp_stream.write_all(connect_request.as_bytes()).await?;
- tcp_stream.flush().await?;
-
- Self::parse_connect_response(&mut tcp_stream).await?;
-
- if is_https {
- Self::secure_stream(tcp_stream, &ssl_connector, &host).await
- } else {
- Ok(MaybeTlsStream::Normal(tcp_stream))
- }
- }.boxed()
- } else {
- async move {
- let tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
-
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
-
- Ok(MaybeTlsStream::Proxied(tcp_stream))
- }.boxed()
- }
- } else {
- async move {
- let dst_str = dst.to_string(); // for error messages
- let tcp_stream = connector
- .call(dst)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", dst_str, err))?;
-
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
-
- if is_https {
- Self::secure_stream(tcp_stream, &ssl_connector, &host).await
- } else {
- Ok(MaybeTlsStream::Normal(tcp_stream))
- }
- }.boxed()
- }
- }
-}
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index 729711c8..fa3eadf4 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -7,10 +7,12 @@ use http::{Request, Response, HeaderValue};
use openssl::ssl::{SslConnector, SslMethod};
use futures::*;
-use proxmox_http::http::ProxyConfig;
+use proxmox_http::http::{
+ ProxyConfig,
+ client::HttpsConnector,
+};
use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
-use crate::tools::http::HttpsConnector;
/// Asyncrounous HTTP client implementation
pub struct SimpleHttp {
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (18 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http Fabian Grünbichler
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
in preparation of moving the abstraction to proxmox_http
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
src/acme/client.rs | 4 +--
src/api2/node/apt.rs | 8 +++--
src/tools.rs | 16 +++++++++
src/tools/simple_http_client.rs | 59 +++++++++++++++++++++------------
src/tools/subscription.rs | 8 +++--
5 files changed, 67 insertions(+), 28 deletions(-)
diff --git a/src/acme/client.rs b/src/acme/client.rs
index 28f277e9..1a6ca46f 100644
--- a/src/acme/client.rs
+++ b/src/acme/client.rs
@@ -19,7 +19,7 @@ use proxmox_acme_rs::{Account, Authorization, Challenge, Directory, Error, Error
use crate::api2::types::AcmeAccountName;
use crate::config::acme::account_path;
-use crate::tools::SimpleHttp;
+use crate::tools::{pbs_simple_http, SimpleHttp};
/// Our on-disk format inherited from PVE's proxmox-acme code.
#[derive(Deserialize, Serialize)]
@@ -72,7 +72,7 @@ impl AcmeClient {
account: None,
directory: None,
nonce: None,
- http_client: SimpleHttp::new(None),
+ http_client: pbs_simple_http(None),
}
}
diff --git a/src/api2/node/apt.rs b/src/api2/node/apt.rs
index 120d5339..9b2d1572 100644
--- a/src/api2/node/apt.rs
+++ b/src/api2/node/apt.rs
@@ -11,7 +11,11 @@ use proxmox_http::http::ProxyConfig;
use crate::config::node;
use crate::server::WorkerTask;
-use crate::tools::{apt, SimpleHttp, subscription};
+use crate::tools::{
+ apt,
+ pbs_simple_http,
+ subscription,
+};
use crate::config::acl::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY};
use crate::api2::types::{Authid, APTUpdateInfo, NODE_SCHEMA, UPID_SCHEMA};
@@ -227,7 +231,7 @@ fn apt_get_changelog(
}
let proxy_config = read_and_update_proxy_config()?;
- let mut client = SimpleHttp::new(proxy_config);
+ let mut client = pbs_simple_http(proxy_config);
let changelog_url = &pkg_info[0].change_log_url;
// FIXME: use 'apt-get changelog' for proxmox packages as well, once repo supports it
diff --git a/src/tools.rs b/src/tools.rs
index eb9a9fd0..4253c054 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -18,6 +18,8 @@ use percent_encoding::{utf8_percent_encode, AsciiSet};
pub use proxmox::tools::fd::Fd;
use proxmox::tools::fs::{create_path, CreateOptions};
+use proxmox_http::http::ProxyConfig;
+
pub mod acl;
pub mod apt;
pub mod async_io;
@@ -34,6 +36,7 @@ pub mod fuse_loop;
mod simple_http_client;
pub use simple_http_client::SimpleHttp;
+pub use simple_http_client::SimpleHttpOptions;
pub mod json;
pub mod logrotate;
@@ -483,6 +486,19 @@ impl<T: Any> AsAny for T {
/// The default 2 hours are far too long for PBS
pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
+pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-backup-client/1.0";
+
+/// Returns a new instance of `SimpleHttp` configured for PBS usage.
+pub fn pbs_simple_http(proxy_config: Option<ProxyConfig>) -> SimpleHttp {
+ let options = SimpleHttpOptions {
+ proxy_config,
+ user_agent: Some(DEFAULT_USER_AGENT_STRING.to_string()),
+ tcp_keepalive: Some(PROXMOX_BACKUP_TCP_KEEPALIVE_TIME),
+ ..Default::default()
+ };
+
+ SimpleHttp::with_options(options)
+}
/// This used to be: `SIMPLE_ENCODE_SET` plus space, `"`, `#`, `<`, `>`, backtick, `?`, `{`, `}`
pub const DEFAULT_ENCODE_SET: &AsciiSet = &percent_encoding::CONTROLS // 0..1f and 7e
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index fa3eadf4..84e9b69f 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -12,50 +12,65 @@ use proxmox_http::http::{
client::HttpsConnector,
};
-use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
+/// Options for a SimpleHttp client.
+#[derive(Default)]
+pub struct SimpleHttpOptions {
+ /// Proxy configuration
+ pub proxy_config: Option<ProxyConfig>,
+ /// `User-Agent` header value, defaults to `proxmox-simple-http-client/0.1`
+ pub user_agent: Option<String>,
+ /// TCP keepalive time, defaults to 7200
+ pub tcp_keepalive: Option<u32>,
+}
+
+impl SimpleHttpOptions {
+ fn get_proxy_authorization(&self) -> Option<String> {
+ if let Some(ref proxy_config) = self.proxy_config {
+ if !proxy_config.force_connect {
+ return proxy_config.authorization.clone();
+ }
+ }
+
+ None
+ }
+}
/// Asyncrounous HTTP client implementation
pub struct SimpleHttp {
client: Client<HttpsConnector, Body>,
- proxy_authorization: Option<String>, // Proxy-Authorization header value
- user_agent: Option<String>,
+ options: SimpleHttpOptions,
}
impl SimpleHttp {
+ pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-simple-http-client/0.1";
- pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-backup-client/1.0";
+ pub fn new() -> Self {
+ Self::with_options(SimpleHttpOptions::default())
+ }
- pub fn new(proxy_config: Option<ProxyConfig>) -> Self {
+ pub fn with_options(options: SimpleHttpOptions) -> Self {
let ssl_connector = SslConnector::builder(SslMethod::tls()).unwrap().build();
- Self::with_ssl_connector(ssl_connector, proxy_config)
+ Self::with_ssl_connector(ssl_connector, options)
}
- pub fn with_ssl_connector(ssl_connector: SslConnector, proxy_config: Option<ProxyConfig>) -> Self {
-
- let mut proxy_authorization = None;
- if let Some(ref proxy_config) = proxy_config {
- if !proxy_config.force_connect {
- proxy_authorization = proxy_config.authorization.clone();
- }
- }
-
+ pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector, PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
- if let Some(proxy_config) = proxy_config {
- https.set_proxy(proxy_config);
+ let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
+ if let Some(ref proxy_config) = options.proxy_config {
+ https.set_proxy(proxy_config.clone());
}
let client = Client::builder().build(https);
- Self { client, proxy_authorization, user_agent: None }
+ Self { client, options }
}
pub fn set_user_agent(&mut self, user_agent: &str) -> Result<(), Error> {
- self.user_agent = Some(user_agent.to_owned());
+ self.options.user_agent = Some(user_agent.to_owned());
Ok(())
}
fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
- if let Some(ref authorization) = self.proxy_authorization {
+ if let Some(ref authorization) = self.options.get_proxy_authorization() {
request
.headers_mut()
.insert(
@@ -68,7 +83,7 @@ impl SimpleHttp {
}
pub async fn request(&self, mut request: Request<Body>) -> Result<Response<Body>, Error> {
- let user_agent = if let Some(ref user_agent) = self.user_agent {
+ let user_agent = if let Some(ref user_agent) = self.options.user_agent {
HeaderValue::from_str(&user_agent)?
} else {
HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
diff --git a/src/tools/subscription.rs b/src/tools/subscription.rs
index 212cf5e8..ed42e630 100644
--- a/src/tools/subscription.rs
+++ b/src/tools/subscription.rs
@@ -7,7 +7,11 @@ use regex::Regex;
use proxmox::api::api;
use crate::config::node;
-use crate::tools::{self, SimpleHttp};
+use crate::tools::{
+ self,
+ pbs_simple_http,
+ SimpleHttp,
+};
use proxmox::tools::fs::{replace_file, CreateOptions};
/// How long the local key is valid for in between remote checks
@@ -109,7 +113,7 @@ async fn register_subscription(
None
};
- let mut client = SimpleHttp::new(proxy_config);
+ let mut client = pbs_simple_http(proxy_config);
let uri = "https://shop.maurer-it.com/modules/servers/licensing/verify.php";
let query = tools::json_object_to_query(params)?;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (19 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #10
src/acme/client.rs | 3 +-
src/tools.rs | 10 +-
src/tools/simple_http_client.rs | 157 --------------------------------
src/tools/subscription.rs | 2 +-
4 files changed, 8 insertions(+), 164 deletions(-)
delete mode 100644 src/tools/simple_http_client.rs
diff --git a/src/acme/client.rs b/src/acme/client.rs
index 1a6ca46f..d1f46617 100644
--- a/src/acme/client.rs
+++ b/src/acme/client.rs
@@ -16,10 +16,11 @@ use proxmox_acme_rs::account::AccountData as AcmeAccountData;
use proxmox_acme_rs::order::{Order, OrderData};
use proxmox_acme_rs::Request as AcmeRequest;
use proxmox_acme_rs::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
+use proxmox_http::http::client::SimpleHttp;
use crate::api2::types::AcmeAccountName;
use crate::config::acme::account_path;
-use crate::tools::{pbs_simple_http, SimpleHttp};
+use crate::tools::pbs_simple_http;
/// Our on-disk format inherited from PVE's proxmox-acme code.
#[derive(Deserialize, Serialize)]
diff --git a/src/tools.rs b/src/tools.rs
index 4253c054..a9f57b88 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -18,7 +18,11 @@ use percent_encoding::{utf8_percent_encode, AsciiSet};
pub use proxmox::tools::fd::Fd;
use proxmox::tools::fs::{create_path, CreateOptions};
-use proxmox_http::http::ProxyConfig;
+use proxmox_http::http::{
+ client::SimpleHttp,
+ client::SimpleHttpOptions,
+ ProxyConfig,
+};
pub mod acl;
pub mod apt;
@@ -34,10 +38,6 @@ pub mod format;
pub mod fs;
pub mod fuse_loop;
-mod simple_http_client;
-pub use simple_http_client::SimpleHttp;
-pub use simple_http_client::SimpleHttpOptions;
-
pub mod json;
pub mod logrotate;
pub mod loopdev;
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
deleted file mode 100644
index 84e9b69f..00000000
--- a/src/tools/simple_http_client.rs
+++ /dev/null
@@ -1,157 +0,0 @@
-use anyhow::{Error, format_err, bail};
-use std::collections::HashMap;
-
-use hyper::Body;
-use hyper::client::{Client, HttpConnector};
-use http::{Request, Response, HeaderValue};
-use openssl::ssl::{SslConnector, SslMethod};
-use futures::*;
-
-use proxmox_http::http::{
- ProxyConfig,
- client::HttpsConnector,
-};
-
-/// Options for a SimpleHttp client.
-#[derive(Default)]
-pub struct SimpleHttpOptions {
- /// Proxy configuration
- pub proxy_config: Option<ProxyConfig>,
- /// `User-Agent` header value, defaults to `proxmox-simple-http-client/0.1`
- pub user_agent: Option<String>,
- /// TCP keepalive time, defaults to 7200
- pub tcp_keepalive: Option<u32>,
-}
-
-impl SimpleHttpOptions {
- fn get_proxy_authorization(&self) -> Option<String> {
- if let Some(ref proxy_config) = self.proxy_config {
- if !proxy_config.force_connect {
- return proxy_config.authorization.clone();
- }
- }
-
- None
- }
-}
-
-/// Asyncrounous HTTP client implementation
-pub struct SimpleHttp {
- client: Client<HttpsConnector, Body>,
- options: SimpleHttpOptions,
-}
-
-impl SimpleHttp {
- pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-simple-http-client/0.1";
-
- pub fn new() -> Self {
- Self::with_options(SimpleHttpOptions::default())
- }
-
- pub fn with_options(options: SimpleHttpOptions) -> Self {
- let ssl_connector = SslConnector::builder(SslMethod::tls()).unwrap().build();
- Self::with_ssl_connector(ssl_connector, options)
- }
-
- pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
- let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
- if let Some(ref proxy_config) = options.proxy_config {
- https.set_proxy(proxy_config.clone());
- }
- let client = Client::builder().build(https);
- Self { client, options }
- }
-
- pub fn set_user_agent(&mut self, user_agent: &str) -> Result<(), Error> {
- self.options.user_agent = Some(user_agent.to_owned());
- Ok(())
- }
-
- fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
- if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
- if let Some(ref authorization) = self.options.get_proxy_authorization() {
- request
- .headers_mut()
- .insert(
- http::header::PROXY_AUTHORIZATION,
- HeaderValue::from_str(authorization)?,
- );
- }
- }
- Ok(())
- }
-
- pub async fn request(&self, mut request: Request<Body>) -> Result<Response<Body>, Error> {
- let user_agent = if let Some(ref user_agent) = self.options.user_agent {
- HeaderValue::from_str(&user_agent)?
- } else {
- HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
- };
-
- request.headers_mut().insert(hyper::header::USER_AGENT, user_agent);
-
- self.add_proxy_headers(&mut request)?;
-
- self.client.request(request)
- .map_err(Error::from)
- .await
- }
-
- pub async fn post(
- &mut self,
- uri: &str,
- body: Option<String>,
- content_type: Option<&str>,
- ) -> Result<Response<Body>, Error> {
-
- let body = if let Some(body) = body {
- Body::from(body)
- } else {
- Body::empty()
- };
- let content_type = content_type.unwrap_or("application/json");
-
- let request = Request::builder()
- .method("POST")
- .uri(uri)
- .header(hyper::header::CONTENT_TYPE, content_type)
- .body(body)?;
-
- self.request(request).await
- }
-
- pub async fn get_string(
- &mut self,
- uri: &str,
- extra_headers: Option<&HashMap<String, String>>,
- ) -> Result<String, Error> {
-
- let mut request = Request::builder()
- .method("GET")
- .uri(uri);
-
- if let Some(hs) = extra_headers {
- for (h, v) in hs.iter() {
- request = request.header(h, v);
- }
- }
-
- let request = request.body(Body::empty())?;
-
- let res = self.request(request).await?;
-
- let status = res.status();
- if !status.is_success() {
- bail!("Got bad status '{}' from server", status)
- }
-
- Self::response_body_string(res).await
- }
-
- pub async fn response_body_string(res: Response<Body>) -> Result<String, Error> {
- let buf = hyper::body::to_bytes(res).await?;
- String::from_utf8(buf.to_vec())
- .map_err(|err| format_err!("Error converting HTTP result data: {}", err))
- }
-}
diff --git a/src/tools/subscription.rs b/src/tools/subscription.rs
index ed42e630..c549c3dd 100644
--- a/src/tools/subscription.rs
+++ b/src/tools/subscription.rs
@@ -10,9 +10,9 @@ use crate::config::node;
use crate::tools::{
self,
pbs_simple_http,
- SimpleHttp,
};
use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_http::http::client::SimpleHttp;
/// How long the local key is valid for in between remote checks
pub const MAX_LOCAL_KEY_AGE: i64 = 15 * 24 * 3600;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] applied: [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
@ 2021-05-17 8:22 ` Dietmar Maurer
0 siblings, 0 replies; 24+ messages in thread
From: Dietmar Maurer @ 2021-05-17 8:22 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
applied all 13 patches
On 5/14/21 3:44 PM, Fabian Grünbichler wrote:
> this is shadowed by the websocket feature being enabled by default.
>
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
>
> Notes:
> fixes issue uncovered by next patch
>
> proxmox/Cargo.toml | 2 +-
> proxmox/debian/control | 1 +
> 2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/proxmox/Cargo.toml b/proxmox/Cargo.toml
> index c04c105..da46e74 100644
> --- a/proxmox/Cargo.toml
> +++ b/proxmox/Cargo.toml
> @@ -57,7 +57,7 @@ sortable-macro = ["proxmox-sortable-macro"]
> api-macro = ["proxmox-api-macro"]
> test-harness = []
> cli = [ "router", "hyper", "tokio" ]
> -router = [ "hyper", "tokio" ]
> +router = [ "futures", "hyper", "tokio" ]
> websocket = [ "futures", "hyper", "openssl", "tokio/sync", "tokio/io-util", "openssl" ]
> tfa = [ "openssl" ]
> u2f = [ "base32" ]
> diff --git a/proxmox/debian/control b/proxmox/debian/control
> index da55b88..9091cb8 100644
> --- a/proxmox/debian/control
> +++ b/proxmox/debian/control
> @@ -267,6 +267,7 @@ Multi-Arch: same
> Depends:
> ${misc:Depends},
> librust-proxmox-dev (= ${binary:Version}),
> + librust-futures-0.3+default-dev,
> librust-hyper-0.14+default-dev,
> librust-hyper-0.14+full-dev,
> librust-tokio-1+default-dev
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] applied: [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
@ 2021-05-17 8:35 ` Dietmar Maurer
0 siblings, 0 replies; 24+ messages in thread
From: Dietmar Maurer @ 2021-05-17 8:35 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
applied all 8 patches
On 5/14/21 3:44 PM, Fabian Grünbichler wrote:
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
>
> Notes:
> requires patch #3 in proxmox
>
> Cargo.toml | 5 +++--
> debian/control | 3 ++-
> src/api2/node.rs | 2 +-
> 3 files changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/Cargo.toml b/Cargo.toml
> index cd99241e..997017c0 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -53,10 +53,11 @@ percent-encoding = "2.1"
> pin-utils = "0.1.0"
> pin-project = "1.0"
> pathpatterns = "0.1.2"
> -proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro", "websocket" ] }
> +proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
> #proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
> -#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro", "websocket" ] }
> +#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
> proxmox-fuse = "0.1.1"
> +proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "websocket" ] }
> pxar = { version = "0.10.1", features = [ "tokio-io" ] }
> #pxar = { path = "../pxar", features = [ "tokio-io" ] }
> regex = "1.2"
> diff --git a/debian/control b/debian/control
> index 3f1edba8..a02518fe 100644
> --- a/debian/control
> +++ b/debian/control
> @@ -42,9 +42,10 @@ Build-Depends: debhelper (>= 11),
> librust-proxmox-0.11+api-macro-dev (>= 0.11.4-~~),
> librust-proxmox-0.11+default-dev (>= 0.11.4-~~),
> librust-proxmox-0.11+sortable-macro-dev (>= 0.11.4-~~),
> - librust-proxmox-0.11+websocket-dev (>= 0.11.4-~~),
> librust-proxmox-acme-rs-0.2+default-dev (>= 0.2.1-~~),
> librust-proxmox-fuse-0.1+default-dev (>= 0.1.1-~~),
> + librust-proxmox-http-0.1+default-dev,
> + librust-proxmox-http-0.1+websocket-dev,
> librust-pxar-0.10+default-dev (>= 0.10.1-~~),
> librust-pxar-0.10+tokio-io-dev (>= 0.10.1-~~),
> librust-regex-1+default-dev (>= 1.2-~~),
> diff --git a/src/api2/node.rs b/src/api2/node.rs
> index 75271cd5..af9ecf69 100644
> --- a/src/api2/node.rs
> +++ b/src/api2/node.rs
> @@ -17,7 +17,7 @@ use proxmox::api::{
> api, schema::*, ApiHandler, ApiMethod, ApiResponseFuture, Permission, RpcEnvironment,
> };
> use proxmox::list_subdirs_api_method;
> -use proxmox::tools::websocket::WebSocket;
> +use proxmox_http::websocket::WebSocket;
> use proxmox::{identity, sortable};
>
> use crate::api2::types::*;
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2021-05-17 8:35 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
2021-05-17 8:22 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 12/13] http: rustfmt Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 13/13] http: update d/control Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
2021-05-17 8:35 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http Fabian Grünbichler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox