* [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http
@ 2021-05-14 13:44 Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
` (20 more replies)
0 siblings, 21 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
a first batch of factoring out HTTP helpers and client code into
a new proxmox_http crate living in the proxmox workspace. the
full-fledged API client ist still in proxmox-backup as it is entagled
with Userid/Authid, and we need to decide whether to move those into
proxmox as well or make the API client generic over some sort of
credentials trait/type/..
proxmox:
Fabian Grünbichler (13):
proxmox: add missing +router -> futures dep
meta: add empty proxmox-http sub-crate
http: takeover websocket feature from proxmox
http: make clippy happy
proxmox: takeover socket helper from proxmox_backup
http: takeover MaybeTlsStream from proxmox_backup
http: takeover build_authority helper from proxmox_backup
http: takeover ProxyConfig from proxmox_backup
http: takeover tools::http from proxmox_backup
http: takeover simple HTTP client from proxmox_backup
http: make clippy happy
http: rustfmt
http: track d/control
21 files changed, 900 insertions(+), 43 deletions(-)
proxmox-backup:
Fabian Grünbichler (8):
move websocket to new 'proxmox_http' crate
refactor: move socket helper to proxmox crate
move MaybeTlsStream wrapper to proxmox_http
HttpsConnector: make keepalive configurable
move ProxyConfig to proxmox_http
move tools::http to proxmox_http
SimpleHttp: factor out product-specific bits
move SimpleHttp to proxmox_http
15 files changed, 58 insertions(+), 585 deletions(-)
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-17 8:22 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate Fabian Grünbichler
` (19 subsequent siblings)
20 siblings, 1 reply; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
this is shadowed by the websocket feature being enabled by default.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
fixes issue uncovered by next patch
proxmox/Cargo.toml | 2 +-
proxmox/debian/control | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/proxmox/Cargo.toml b/proxmox/Cargo.toml
index c04c105..da46e74 100644
--- a/proxmox/Cargo.toml
+++ b/proxmox/Cargo.toml
@@ -57,7 +57,7 @@ sortable-macro = ["proxmox-sortable-macro"]
api-macro = ["proxmox-api-macro"]
test-harness = []
cli = [ "router", "hyper", "tokio" ]
-router = [ "hyper", "tokio" ]
+router = [ "futures", "hyper", "tokio" ]
websocket = [ "futures", "hyper", "openssl", "tokio/sync", "tokio/io-util", "openssl" ]
tfa = [ "openssl" ]
u2f = [ "base32" ]
diff --git a/proxmox/debian/control b/proxmox/debian/control
index da55b88..9091cb8 100644
--- a/proxmox/debian/control
+++ b/proxmox/debian/control
@@ -267,6 +267,7 @@ Multi-Arch: same
Depends:
${misc:Depends},
librust-proxmox-dev (= ${binary:Version}),
+ librust-futures-0.3+default-dev,
librust-hyper-0.14+default-dev,
librust-hyper-0.14+full-dev,
librust-tokio-1+default-dev
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox Fabian Grünbichler
` (18 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Makefile | 2 +-
Cargo.toml | 1 +
proxmox-http/Cargo.toml | 20 +++
proxmox-http/debian/changelog | 5 +
proxmox-http/debian/control | 230 ++++++++++++++++++++++++++++++
proxmox-http/debian/copyright | 16 +++
proxmox-http/debian/debcargo.toml | 7 +
proxmox-http/src/lib.rs | 0
8 files changed, 280 insertions(+), 1 deletion(-)
create mode 100644 proxmox-http/Cargo.toml
create mode 100644 proxmox-http/debian/changelog
create mode 100644 proxmox-http/debian/control
create mode 100644 proxmox-http/debian/copyright
create mode 100644 proxmox-http/debian/debcargo.toml
create mode 100644 proxmox-http/src/lib.rs
diff --git a/Makefile b/Makefile
index ebaf113..4f74539 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
# Shortcut for common operations:
-CRATES=proxmox proxmox-api-macro proxmox-sortable-macro
+CRATES=proxmox proxmox-api-macro proxmox-http proxmox-sortable-macro
# By default we just run checks:
.PHONY: all
diff --git a/Cargo.toml b/Cargo.toml
index d6dc7b4..12e832d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -2,6 +2,7 @@
members = [
"proxmox",
"proxmox-api-macro",
+ "proxmox-http",
"proxmox-sortable-macro",
]
exclude = [
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
new file mode 100644
index 0000000..9c2d3c5
--- /dev/null
+++ b/proxmox-http/Cargo.toml
@@ -0,0 +1,20 @@
+[package]
+name = "proxmox-http"
+edition = "2018"
+version = "0.1.0"
+authors = [
+ "Dietmar Maurer <dietmar@proxmox.com>",
+ "Wolfgang Bumiller <w.bumiller@proxmox.com>",
+]
+license = "AGPL-3"
+description = "Proxmox HTTP library"
+
+exclude = [ "debian" ]
+
+[dependencies]
+
+[features]
+default = []
+
+client = []
+websocket = []
diff --git a/proxmox-http/debian/changelog b/proxmox-http/debian/changelog
new file mode 100644
index 0000000..2ef333f
--- /dev/null
+++ b/proxmox-http/debian/changelog
@@ -0,0 +1,5 @@
+rust-proxmox-http (0.1.0-1) unstable; urgency=medium
+
+ * refactor code from proxmox-backup and proxmox into own HTTP crate
+
+ -- Proxmox Support Team <support@proxmox.com> Tue, 11 May 2021 10:50:29 +0200
diff --git a/proxmox-http/debian/control b/proxmox-http/debian/control
new file mode 100644
index 0000000..f010d6e
--- /dev/null
+++ b/proxmox-http/debian/control
@@ -0,0 +1,230 @@
+Source: rust-proxmox-http
+Section: rust
+Priority: optional
+Build-Depends: debhelper (>= 11),
+ dh-cargo (>= 18),
+ cargo:native <!nocheck>,
+ rustc:native <!nocheck>,
+ libstd-rust-dev <!nocheck>,
+ librust-anyhow-1+default-dev <!nocheck>
+Maintainer: Proxmox Support Team <support@proxmox.com>
+Standards-Version: 4.4.1
+Vcs-Git: git://git.proxmox.com/git/proxmox.git
+Vcs-Browser: https://git.proxmox.com/?p=proxmox.git
+
+Package: librust-proxmox-http-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-anyhow-1+default-dev
+Suggests:
+ librust-proxmox-http+base64-dev (= ${binary:Version}),
+ librust-proxmox-http+client-dev (= ${binary:Version}),
+ librust-proxmox-http+futures-dev (= ${binary:Version}),
+ librust-proxmox-http+http-dev (= ${binary:Version}),
+ librust-proxmox-http+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http+hyper-dev (= ${binary:Version}),
+ librust-proxmox-http+openssl-dev (= ${binary:Version}),
+ librust-proxmox-http+proxmox-dev (= ${binary:Version}),
+ librust-proxmox-http+tokio-dev (= ${binary:Version}),
+ librust-proxmox-http+tokio-openssl-dev (= ${binary:Version}),
+ librust-proxmox-http+websocket-dev (= ${binary:Version})
+Provides:
+ librust-proxmox-http+default-dev (= ${binary:Version}),
+ librust-proxmox-http-0-dev (= ${binary:Version}),
+ librust-proxmox-http-0+default-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+default-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+default-dev (= ${binary:Version})
+Description: Proxmox HTTP library - Rust source code
+ This package contains the source for the Rust proxmox-http crate, packaged by
+ debcargo for use with cargo and dh-cargo.
+
+Package: librust-proxmox-http+base64-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-base64-0.12+default-dev
+Provides:
+ librust-proxmox-http-0+base64-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+base64-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+base64-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "base64"
+ This metapackage enables feature "base64" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+client-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-proxmox-http+http-helpers-dev (= ${binary:Version}),
+ librust-futures-0.3+default-dev,
+ librust-openssl-0.10+default-dev
+Provides:
+ librust-proxmox-http-0+client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+client-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "client"
+ This metapackage enables feature "client" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+futures-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-futures-0.3+default-dev
+Provides:
+ librust-proxmox-http-0+futures-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+futures-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+futures-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "futures"
+ This metapackage enables feature "futures" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+http-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-http-0.2+default-dev
+Provides:
+ librust-proxmox-http-0+http-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+http-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+http-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "http"
+ This metapackage enables feature "http" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+http-helpers-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-base64-0.12+default-dev,
+ librust-http-0.2+default-dev,
+ librust-hyper-0.14+default-dev,
+ librust-hyper-0.14+full-dev,
+ librust-proxmox-0.11-dev (>= 0.11.3-~~),
+ librust-tokio-1+io-util-dev,
+ librust-tokio-openssl-0.6+default-dev (>= 0.6.1-~~)
+Provides:
+ librust-proxmox-http+simple-client-dev (= ${binary:Version}),
+ librust-proxmox-http-0+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http-0+simple-client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+simple-client-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+http-helpers-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+simple-client-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "http-helpers" and 1 more
+ This metapackage enables feature "http-helpers" for the Rust proxmox-http
+ crate, by pulling in any additional dependencies needed by that feature.
+ .
+ Additionally, this package also provides the "simple-client" feature.
+
+Package: librust-proxmox-http+hyper-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-hyper-0.14+default-dev,
+ librust-hyper-0.14+full-dev
+Provides:
+ librust-proxmox-http-0+hyper-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+hyper-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+hyper-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "hyper"
+ This metapackage enables feature "hyper" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+openssl-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-openssl-0.10+default-dev
+Provides:
+ librust-proxmox-http-0+openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+openssl-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "openssl"
+ This metapackage enables feature "openssl" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+proxmox-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-proxmox-0.11-dev (>= 0.11.3-~~)
+Provides:
+ librust-proxmox-http-0+proxmox-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+proxmox-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+proxmox-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "proxmox"
+ This metapackage enables feature "proxmox" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+tokio-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-tokio-1+default-dev
+Provides:
+ librust-proxmox-http-0+tokio-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+tokio-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+tokio-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "tokio"
+ This metapackage enables feature "tokio" for the Rust proxmox-http crate, by
+ pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+tokio-openssl-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-tokio-openssl-0.6+default-dev (>= 0.6.1-~~)
+Provides:
+ librust-proxmox-http-0+tokio-openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+tokio-openssl-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+tokio-openssl-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "tokio-openssl"
+ This metapackage enables feature "tokio-openssl" for the Rust proxmox-http
+ crate, by pulling in any additional dependencies needed by that feature.
+
+Package: librust-proxmox-http+websocket-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-proxmox-http-dev (= ${binary:Version}),
+ librust-base64-0.12+default-dev,
+ librust-futures-0.3+default-dev,
+ librust-hyper-0.14+default-dev,
+ librust-hyper-0.14+full-dev,
+ librust-openssl-0.10+default-dev,
+ librust-proxmox-0.11+tokio-dev (>= 0.11.3-~~),
+ librust-tokio-1+io-util-dev,
+ librust-tokio-1+sync-dev
+Provides:
+ librust-proxmox-http-0+websocket-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1+websocket-dev (= ${binary:Version}),
+ librust-proxmox-http-0.1.0+websocket-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "websocket"
+ This metapackage enables feature "websocket" for the Rust proxmox-http crate,
+ by pulling in any additional dependencies needed by that feature.
diff --git a/proxmox-http/debian/copyright b/proxmox-http/debian/copyright
new file mode 100644
index 0000000..2c91d38
--- /dev/null
+++ b/proxmox-http/debian/copyright
@@ -0,0 +1,16 @@
+Copyright (C) 2019,2020 Proxmox Server Solutions GmbH
+
+This software is written by Proxmox Server Solutions GmbH <support@proxmox.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
diff --git a/proxmox-http/debian/debcargo.toml b/proxmox-http/debian/debcargo.toml
new file mode 100644
index 0000000..b7864cd
--- /dev/null
+++ b/proxmox-http/debian/debcargo.toml
@@ -0,0 +1,7 @@
+overlay = "."
+crate_src_path = ".."
+maintainer = "Proxmox Support Team <support@proxmox.com>"
+
+[source]
+vcs_git = "git://git.proxmox.com/git/proxmox.git"
+vcs_browser = "https://git.proxmox.com/?p=proxmox.git"
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
new file mode 100644
index 0000000..e69de29
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy Fabian Grünbichler
` (17 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
adapted: use statements for proxmox::*, use statements for doctests
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
BREAKING change! requires corresponding update in proxmox-backup
proxmox-http/Cargo.toml | 10 ++++++-
proxmox-http/src/lib.rs | 2 ++
.../src/websocket/mod.rs | 14 ++++-----
proxmox/Cargo.toml | 3 +-
proxmox/debian/control | 29 ++-----------------
proxmox/src/tools/mod.rs | 3 --
6 files changed, 21 insertions(+), 40 deletions(-)
rename proxmox/src/tools/websocket.rs => proxmox-http/src/websocket/mod.rs (98%)
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 9c2d3c5..641ace2 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -12,9 +12,17 @@ description = "Proxmox HTTP library"
exclude = [ "debian" ]
[dependencies]
+anyhow = "1.0"
+base64 = { version = "0.12", optional = true }
+futures = { version = "0.3", optional = true }
+hyper = { version = "0.14", features = [ "full" ], optional = true }
+openssl = { version = "0.10", optional = true }
+tokio = { version = "1.0", features = [], optional = true }
+
+proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-features = false }
[features]
default = []
client = []
-websocket = []
+websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
index e69de29..31e44af 100644
--- a/proxmox-http/src/lib.rs
+++ b/proxmox-http/src/lib.rs
@@ -0,0 +1,2 @@
+#[cfg(feature = "websocket")]
+pub mod websocket;
diff --git a/proxmox/src/tools/websocket.rs b/proxmox-http/src/websocket/mod.rs
similarity index 98%
rename from proxmox/src/tools/websocket.rs
rename to proxmox-http/src/websocket/mod.rs
index 83679b6..aac7304 100644
--- a/proxmox/src/tools/websocket.rs
+++ b/proxmox-http/src/websocket/mod.rs
@@ -22,8 +22,8 @@ use tokio::sync::mpsc;
use futures::future::FutureExt;
use futures::ready;
-use crate::sys::error::io_err_other;
-use crate::tools::byte_buffer::ByteBuffer;
+use proxmox::sys::error::io_err_other;
+use proxmox::tools::byte_buffer::ByteBuffer;
// see RFC6455 section 7.4.1
#[derive(Debug, Clone, Copy)]
@@ -146,7 +146,7 @@ fn mask_bytes(mask: Option<[u8; 4]>, data: &mut [u8]) {
///
/// A normal Frame
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let data = vec![0,1,2,3,4];
@@ -159,7 +159,7 @@ fn mask_bytes(mask: Option<[u8; 4]>, data: &mut [u8]) {
///
/// A masked Frame
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let data = vec![0,1,2,3,4];
@@ -172,7 +172,7 @@ fn mask_bytes(mask: Option<[u8; 4]>, data: &mut [u8]) {
///
/// A ping Frame
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let data = vec![0,1,2,3,4];
@@ -233,7 +233,7 @@ pub fn create_frame(
///
/// Example usage:
/// ```
-/// # use proxmox::tools::websocket::*;
+/// # use proxmox_http::websocket::*;
/// # use std::io;
/// # use tokio::io::{AsyncWrite, AsyncWriteExt};
/// async fn code<I: AsyncWrite + Unpin>(writer: I) -> io::Result<()> {
@@ -352,7 +352,7 @@ impl FrameHeader {
///
/// Example:
/// ```
- /// # use proxmox::tools::websocket::*;
+ /// # use proxmox_http::websocket::*;
/// # use std::io;
/// # fn main() -> Result<(), WebSocketError> {
/// let frame = create_frame(None, &[0,1,2,3], OpCode::Ping)?;
diff --git a/proxmox/Cargo.toml b/proxmox/Cargo.toml
index da46e74..5698969 100644
--- a/proxmox/Cargo.toml
+++ b/proxmox/Cargo.toml
@@ -50,7 +50,7 @@ proxmox-api-macro = { path = "../proxmox-api-macro", optional = true, version =
proxmox-sortable-macro = { path = "../proxmox-sortable-macro", optional = true, version = "0.1.1" }
[features]
-default = [ "cli", "router", "tfa", "u2f", "websocket" ]
+default = [ "cli", "router", "tfa", "u2f" ]
sortable-macro = ["proxmox-sortable-macro"]
# api:
@@ -58,7 +58,6 @@ api-macro = ["proxmox-api-macro"]
test-harness = []
cli = [ "router", "hyper", "tokio" ]
router = [ "futures", "hyper", "tokio" ]
-websocket = [ "futures", "hyper", "openssl", "tokio/sync", "tokio/io-util", "openssl" ]
tfa = [ "openssl" ]
u2f = [ "base32" ]
diff --git a/proxmox/debian/control b/proxmox/debian/control
index 9091cb8..0ba2bc9 100644
--- a/proxmox/debian/control
+++ b/proxmox/debian/control
@@ -28,15 +28,12 @@ Build-Depends: debhelper (>= 11),
librust-serde-json-1+default-dev <!nocheck>,
librust-textwrap-0.11+default-dev <!nocheck>,
librust-tokio-1+default-dev <!nocheck>,
- librust-tokio-1+io-util-dev <!nocheck>,
- librust-tokio-1+sync-dev <!nocheck>,
librust-url-2+default-dev (>= 2.1-~~) <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
uuid-dev <!nocheck>,
- uuid-dev <!nocheck>,
uuid-dev <!nocheck>
Maintainer: Proxmox Support Team <support@proxmox.com>
Standards-Version: 4.4.1
@@ -79,8 +76,7 @@ Suggests:
librust-proxmox+proxmox-sortable-macro-dev (= ${binary:Version}),
librust-proxmox+router-dev (= ${binary:Version}),
librust-proxmox+tokio-dev (= ${binary:Version}),
- librust-proxmox+tokio-stream-dev (= ${binary:Version}),
- librust-proxmox+websocket-dev (= ${binary:Version})
+ librust-proxmox+tokio-stream-dev (= ${binary:Version})
Provides:
librust-proxmox+test-harness-dev (= ${binary:Version}),
librust-proxmox-0-dev (= ${binary:Version}),
@@ -162,8 +158,7 @@ Depends:
librust-proxmox+cli-dev (= ${binary:Version}),
librust-proxmox+router-dev (= ${binary:Version}),
librust-proxmox+tfa-dev (= ${binary:Version}),
- librust-proxmox+u2f-dev (= ${binary:Version}),
- librust-proxmox+websocket-dev (= ${binary:Version})
+ librust-proxmox+u2f-dev (= ${binary:Version})
Provides:
librust-proxmox-0+default-dev (= ${binary:Version}),
librust-proxmox-0.11+default-dev (= ${binary:Version}),
@@ -308,23 +303,3 @@ Provides:
Description: Proxmox library - feature "tokio-stream"
This metapackage enables feature "tokio-stream" for the Rust proxmox crate, by
pulling in any additional dependencies needed by that feature.
-
-Package: librust-proxmox+websocket-dev
-Architecture: any
-Multi-Arch: same
-Depends:
- ${misc:Depends},
- librust-proxmox-dev (= ${binary:Version}),
- librust-futures-0.3+default-dev,
- librust-hyper-0.14+default-dev,
- librust-hyper-0.14+full-dev,
- librust-openssl-0.10+default-dev,
- librust-tokio-1+io-util-dev,
- librust-tokio-1+sync-dev
-Provides:
- librust-proxmox-0+websocket-dev (= ${binary:Version}),
- librust-proxmox-0.11+websocket-dev (= ${binary:Version}),
- librust-proxmox-0.11.4+websocket-dev (= ${binary:Version})
-Description: Proxmox library - feature "websocket"
- This metapackage enables feature "websocket" for the Rust proxmox crate, by
- pulling in any additional dependencies needed by that feature.
diff --git a/proxmox/src/tools/mod.rs b/proxmox/src/tools/mod.rs
index 45f46f9..b172b09 100644
--- a/proxmox/src/tools/mod.rs
+++ b/proxmox/src/tools/mod.rs
@@ -21,9 +21,6 @@ pub mod time;
pub mod uuid;
pub mod vec;
-#[cfg(feature = "websocket")]
-pub mod websocket;
-
#[cfg(feature = "tfa")]
pub mod tfa;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (2 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup Fabian Grünbichler
` (16 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/websocket/mod.rs | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/proxmox-http/src/websocket/mod.rs b/proxmox-http/src/websocket/mod.rs
index aac7304..b8f31dc 100644
--- a/proxmox-http/src/websocket/mod.rs
+++ b/proxmox-http/src/websocket/mod.rs
@@ -202,8 +202,7 @@ pub fn create_frame(
0b00000000
};
- let mut buf = Vec::new();
- buf.push(first_byte);
+ let mut buf = vec![first_byte];
if len < 126 {
buf.push(mask_bit | (len as u8));
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (3 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream " Fabian Grünbichler
` (15 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox/src/sys/linux/mod.rs | 1 +
proxmox/src/sys/linux/socket.rs | 21 +++++++++++++++++++++
2 files changed, 22 insertions(+)
create mode 100644 proxmox/src/sys/linux/socket.rs
diff --git a/proxmox/src/sys/linux/mod.rs b/proxmox/src/sys/linux/mod.rs
index ee616a3..ccd710c 100644
--- a/proxmox/src/sys/linux/mod.rs
+++ b/proxmox/src/sys/linux/mod.rs
@@ -6,6 +6,7 @@ pub mod magic;
pub mod pid;
pub mod procfs;
pub mod pty;
+pub mod socket;
pub mod tty;
/// Get pseudo random data (/dev/urandom)
diff --git a/proxmox/src/sys/linux/socket.rs b/proxmox/src/sys/linux/socket.rs
new file mode 100644
index 0000000..6c63c3c
--- /dev/null
+++ b/proxmox/src/sys/linux/socket.rs
@@ -0,0 +1,21 @@
+use std::os::unix::io::RawFd;
+
+use nix::sys::socket::sockopt::{KeepAlive, TcpKeepIdle};
+use nix::sys::socket::setsockopt;
+
+/// Set TCP keepalive time on a socket
+///
+/// See "man 7 tcp" for details.
+///
+/// The default on Linux is 7200 (2 hours) which is far too long for
+/// many of our use cases.
+pub fn set_tcp_keepalive(
+ socket_fd: RawFd,
+ tcp_keepalive_time: u32,
+) -> nix::Result<()> {
+
+ setsockopt(socket_fd, KeepAlive, &true)?;
+ setsockopt(socket_fd, TcpKeepIdle, &tcp_keepalive_time)?;
+
+ Ok(())
+}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (4 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper " Fabian Grünbichler
` (14 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
this is just a (rather HTTP specific) wrapper, so put it into a
'wrapper' module for now.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 4 +-
proxmox-http/src/http/mod.rs | 3 +
proxmox-http/src/http/wrapper.rs | 122 +++++++++++++++++++++++++++++++
proxmox-http/src/lib.rs | 3 +
4 files changed, 131 insertions(+), 1 deletion(-)
create mode 100644 proxmox-http/src/http/mod.rs
create mode 100644 proxmox-http/src/http/wrapper.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 641ace2..f1f53da 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -18,11 +18,13 @@ futures = { version = "0.3", optional = true }
hyper = { version = "0.14", features = [ "full" ], optional = true }
openssl = { version = "0.10", optional = true }
tokio = { version = "1.0", features = [], optional = true }
+tokio-openssl = { version = "0.6.1", optional = true }
proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-features = false }
[features]
default = []
-client = []
+client = [ "http-helpers" ]
+http-helpers = [ "hyper", "tokio/io-util", "tokio-openssl" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
new file mode 100644
index 0000000..09fa42f
--- /dev/null
+++ b/proxmox-http/src/http/mod.rs
@@ -0,0 +1,3 @@
+mod wrapper;
+
+pub use wrapper::MaybeTlsStream;
diff --git a/proxmox-http/src/http/wrapper.rs b/proxmox-http/src/http/wrapper.rs
new file mode 100644
index 0000000..3399b28
--- /dev/null
+++ b/proxmox-http/src/http/wrapper.rs
@@ -0,0 +1,122 @@
+use std::io;
+use std::pin::Pin;
+use std::task::{Context, Poll};
+
+use hyper::client::connect::{Connection, Connected};
+use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
+use tokio_openssl::SslStream;
+
+/// Asynchronous stream, possibly encrypted and proxied
+///
+/// Usefule for HTTP client implementations using hyper.
+pub enum MaybeTlsStream<S> {
+ Normal(S),
+ Proxied(S),
+ Secured(SslStream<S>),
+}
+
+impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
+ fn poll_read(
+ self: Pin<&mut Self>,
+ cx: &mut Context,
+ buf: &mut ReadBuf,
+ ) -> Poll<Result<(), io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_read(cx, buf)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_read(cx, buf)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_read(cx, buf)
+ }
+ }
+ }
+}
+
+impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
+ fn poll_write(
+ self: Pin<&mut Self>,
+ cx: &mut Context,
+ buf: &[u8],
+ ) -> Poll<Result<usize, io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_write(cx, buf)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_write(cx, buf)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_write(cx, buf)
+ }
+ }
+ }
+
+ fn poll_write_vectored(
+ self: Pin<&mut Self>,
+ cx: &mut Context<'_>,
+ bufs: &[io::IoSlice<'_>],
+ ) -> Poll<Result<usize, io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_write_vectored(cx, bufs)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_write_vectored(cx, bufs)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_write_vectored(cx, bufs)
+ }
+ }
+ }
+
+ fn is_write_vectored(&self) -> bool {
+ match self {
+ MaybeTlsStream::Normal(s) => s.is_write_vectored(),
+ MaybeTlsStream::Proxied(s) => s.is_write_vectored(),
+ MaybeTlsStream::Secured(s) => s.is_write_vectored(),
+ }
+ }
+
+ fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_flush(cx)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_flush(cx)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_flush(cx)
+ }
+ }
+ }
+
+ fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
+ match self.get_mut() {
+ MaybeTlsStream::Normal(ref mut s) => {
+ Pin::new(s).poll_shutdown(cx)
+ }
+ MaybeTlsStream::Proxied(ref mut s) => {
+ Pin::new(s).poll_shutdown(cx)
+ }
+ MaybeTlsStream::Secured(ref mut s) => {
+ Pin::new(s).poll_shutdown(cx)
+ }
+ }
+ }
+}
+
+// we need this for the hyper http client
+impl <S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S>
+{
+ fn connected(&self) -> Connected {
+ match self {
+ MaybeTlsStream::Normal(s) => s.connected(),
+ MaybeTlsStream::Proxied(s) => s.connected().proxy(true),
+ MaybeTlsStream::Secured(s) => s.get_ref().connected(),
+ }
+ }
+}
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
index 31e44af..5e3f9ec 100644
--- a/proxmox-http/src/lib.rs
+++ b/proxmox-http/src/lib.rs
@@ -1,2 +1,5 @@
#[cfg(feature = "websocket")]
pub mod websocket;
+
+#[cfg(feature = "http-helpers")]
+pub mod http;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (5 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig " Fabian Grünbichler
` (13 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 3 ++-
proxmox-http/src/http/helpers.rs | 15 +++++++++++++++
proxmox-http/src/http/mod.rs | 2 ++
3 files changed, 19 insertions(+), 1 deletion(-)
create mode 100644 proxmox-http/src/http/helpers.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index f1f53da..6b2d8db 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -15,6 +15,7 @@ exclude = [ "debian" ]
anyhow = "1.0"
base64 = { version = "0.12", optional = true }
futures = { version = "0.3", optional = true }
+http = { version = "0.2", optional = true }
hyper = { version = "0.14", features = [ "full" ], optional = true }
openssl = { version = "0.10", optional = true }
tokio = { version = "1.0", features = [], optional = true }
@@ -26,5 +27,5 @@ proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-fe
default = []
client = [ "http-helpers" ]
-http-helpers = [ "hyper", "tokio/io-util", "tokio-openssl" ]
+http-helpers = [ "http", "hyper", "tokio/io-util", "tokio-openssl" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/helpers.rs b/proxmox-http/src/http/helpers.rs
new file mode 100644
index 0000000..3f663d2
--- /dev/null
+++ b/proxmox-http/src/http/helpers.rs
@@ -0,0 +1,15 @@
+use anyhow::Error;
+
+use http::uri::Authority;
+
+// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
+pub fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
+ let bytes = host.as_bytes();
+ let len = bytes.len();
+ let authority = if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len-1] != b']' {
+ format!("[{}]:{}", host, port).parse()?
+ } else {
+ format!("{}:{}", host, port).parse()?
+ };
+ Ok(authority)
+}
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
index 09fa42f..4960246 100644
--- a/proxmox-http/src/http/mod.rs
+++ b/proxmox-http/src/http/mod.rs
@@ -1,3 +1,5 @@
mod wrapper;
pub use wrapper::MaybeTlsStream;
+
+pub mod helpers;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (6 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http " Fabian Grünbichler
` (12 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
adapted to use moved build_authority helper.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 2 +-
proxmox-http/src/http/mod.rs | 4 +-
proxmox-http/src/http/proxy_config.rs | 84 +++++++++++++++++++++++++++
3 files changed, 88 insertions(+), 2 deletions(-)
create mode 100644 proxmox-http/src/http/proxy_config.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 6b2d8db..9c6fc35 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -27,5 +27,5 @@ proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-fe
default = []
client = [ "http-helpers" ]
-http-helpers = [ "http", "hyper", "tokio/io-util", "tokio-openssl" ]
+http-helpers = [ "base64", "http", "hyper", "tokio/io-util", "tokio-openssl", "proxmox" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
index 4960246..055648e 100644
--- a/proxmox-http/src/http/mod.rs
+++ b/proxmox-http/src/http/mod.rs
@@ -1,5 +1,7 @@
mod wrapper;
-
pub use wrapper::MaybeTlsStream;
pub mod helpers;
+
+mod proxy_config;
+pub use proxy_config::ProxyConfig;
diff --git a/proxmox-http/src/http/proxy_config.rs b/proxmox-http/src/http/proxy_config.rs
new file mode 100644
index 0000000..7e93a47
--- /dev/null
+++ b/proxmox-http/src/http/proxy_config.rs
@@ -0,0 +1,84 @@
+use anyhow::{Error, format_err, bail};
+
+use http::Uri;
+
+use crate::http::helpers;
+
+/// HTTP Proxy Configuration
+#[derive(Clone)]
+pub struct ProxyConfig {
+ pub host: String,
+ pub port: u16,
+ pub authorization: Option<String>, // user:pass
+ pub force_connect: bool,
+}
+
+impl ProxyConfig {
+ /// Parse proxy config from ALL_PROXY environment var
+ pub fn from_proxy_env() -> Result<Option<ProxyConfig>, Error> {
+
+ // We only support/use ALL_PROXY environment
+
+ match std::env::var_os("ALL_PROXY") {
+ None => return Ok(None),
+ Some(all_proxy) => {
+ let all_proxy = match all_proxy.to_str() {
+ Some(s) => String::from(s),
+ None => bail!("non UTF-8 content in env ALL_PROXY"),
+ };
+ if all_proxy.is_empty() {
+ return Ok(None);
+ }
+ let config = Self::parse_proxy_url(&all_proxy)?;
+ Ok(Some(config))
+ }
+ }
+ }
+
+ /// Parse proxy configuration string [http://]<host>[:port]
+ ///
+ /// Default port is 1080 (like curl)
+ pub fn parse_proxy_url(http_proxy: &str) -> Result<ProxyConfig, Error> {
+ proxmox::try_block!({
+ let proxy_uri: Uri = http_proxy.parse()?;
+ let proxy_authority = match proxy_uri.authority() {
+ Some(authority) => authority,
+ None => bail!("missing proxy authority"),
+ };
+ let host = proxy_authority.host().to_owned();
+ let port = match proxy_uri.port() {
+ Some(port) => port.as_u16(),
+ None => 1080, // CURL default port
+ };
+
+ match proxy_uri.scheme_str() {
+ Some("http") => { /* Ok */ }
+ Some(scheme) => bail!("unsupported proxy scheme '{}'", scheme),
+ None => { /* assume HTTP */ }
+ }
+
+ let authority_vec: Vec<&str> = proxy_authority.as_str().rsplitn(2, '@').collect();
+ let authorization = if authority_vec.len() == 2 {
+ Some(authority_vec[1].to_string())
+ } else {
+ None
+ };
+
+ Ok(ProxyConfig {
+ host,
+ port,
+ authorization,
+ force_connect: false,
+ })
+ }).map_err(|err| format_err!("parse_proxy_url failed: {}", err))
+ }
+
+ /// Assemble canonical proxy string (including scheme and port)
+ pub fn to_proxy_string(&self) -> Result<String, Error> {
+ let authority = helpers::build_authority(&self.host, self.port)?;
+ Ok(match self.authorization {
+ None => format!("http://{}", authority),
+ Some(ref authorization) => format!("http://{}@{}", authorization, authority)
+ })
+ }
+}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (7 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client " Fabian Grünbichler
` (11 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
the parts that were not already moved.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/Cargo.toml | 2 +-
proxmox-http/src/http/client.rs | 3 +
proxmox-http/src/http/client/connector.rs | 203 ++++++++++++++++++++++
proxmox-http/src/http/mod.rs | 3 +
proxmox-http/src/lib.rs | 2 +-
5 files changed, 211 insertions(+), 2 deletions(-)
create mode 100644 proxmox-http/src/http/client.rs
create mode 100644 proxmox-http/src/http/client/connector.rs
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index 9c6fc35..cc3eef6 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -26,6 +26,6 @@ proxmox = { path = "../proxmox", optional = true, version = "0.11.3", default-fe
[features]
default = []
-client = [ "http-helpers" ]
+client = [ "futures", "http-helpers", "openssl" ]
http-helpers = [ "base64", "http", "hyper", "tokio/io-util", "tokio-openssl", "proxmox" ]
websocket = [ "base64", "futures", "hyper", "openssl", "proxmox/tokio", "tokio/io-util", "tokio/sync" ]
diff --git a/proxmox-http/src/http/client.rs b/proxmox-http/src/http/client.rs
new file mode 100644
index 0000000..21a65e3
--- /dev/null
+++ b/proxmox-http/src/http/client.rs
@@ -0,0 +1,3 @@
+mod connector;
+
+pub use connector::HttpsConnector;
diff --git a/proxmox-http/src/http/client/connector.rs b/proxmox-http/src/http/client/connector.rs
new file mode 100644
index 0000000..a302dd1
--- /dev/null
+++ b/proxmox-http/src/http/client/connector.rs
@@ -0,0 +1,203 @@
+use anyhow::{Error, format_err, bail};
+use std::os::unix::io::AsRawFd;
+use std::pin::Pin;
+use std::sync::Arc;
+use std::task::{Context, Poll};
+
+use futures::*;
+use http::Uri;
+use hyper::client::HttpConnector;
+use openssl::ssl::SslConnector;
+use tokio::io::{
+ AsyncRead,
+ AsyncReadExt,
+ AsyncWriteExt,
+};
+use tokio::net::TcpStream;
+use tokio_openssl::SslStream;
+
+use proxmox::sys::linux::socket::set_tcp_keepalive;
+
+use crate::http::{helpers, MaybeTlsStream, ProxyConfig};
+
+#[derive(Clone)]
+pub struct HttpsConnector {
+ connector: HttpConnector,
+ ssl_connector: Arc<SslConnector>,
+ proxy: Option<ProxyConfig>,
+ tcp_keepalive: u32,
+}
+
+impl HttpsConnector {
+ pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
+ connector.enforce_http(false);
+ Self {
+ connector,
+ ssl_connector: Arc::new(ssl_connector),
+ proxy: None,
+ tcp_keepalive,
+ }
+ }
+
+ pub fn set_proxy(&mut self, proxy: ProxyConfig) {
+ self.proxy = Some(proxy);
+ }
+
+ async fn secure_stream(
+ tcp_stream: TcpStream,
+ ssl_connector: &SslConnector,
+ host: &str,
+ ) -> Result<MaybeTlsStream<TcpStream>, Error> {
+ let config = ssl_connector.configure()?;
+ let mut conn: SslStream<TcpStream> = SslStream::new(config.into_ssl(host)?, tcp_stream)?;
+ Pin::new(&mut conn).connect().await?;
+ Ok(MaybeTlsStream::Secured(conn))
+ }
+
+ fn parse_status_line(status_line: &str) -> Result<(), Error> {
+ if !(status_line.starts_with("HTTP/1.1 200") || status_line.starts_with("HTTP/1.0 200")) {
+ bail!("proxy connect failed - invalid status: {}", status_line)
+ }
+ Ok(())
+ }
+
+ async fn parse_connect_response<R: AsyncRead + Unpin>(
+ stream: &mut R,
+ ) -> Result<(), Error> {
+
+ let mut data: Vec<u8> = Vec::new();
+ let mut buffer = [0u8; 256];
+ const END_MARK: &[u8; 4] = b"\r\n\r\n";
+
+ 'outer: loop {
+ let n = stream.read(&mut buffer[..]).await?;
+ if n == 0 { break; }
+ let search_start = if data.len() > END_MARK.len() { data.len() - END_MARK.len() + 1 } else { 0 };
+ data.extend(&buffer[..n]);
+ if data.len() >= END_MARK.len() {
+ if let Some(pos) = data[search_start..].windows(END_MARK.len()).position(|w| w == END_MARK) {
+ let response = String::from_utf8_lossy(&data);
+ let status_line = match response.split("\r\n").next() {
+ Some(status) => status,
+ None => bail!("missing newline"),
+ };
+ Self::parse_status_line(status_line)?;
+
+ if pos != data.len() - END_MARK.len() {
+ bail!("unexpected data after connect response");
+ }
+ break 'outer;
+ }
+ }
+ if data.len() > 1024*32 { // max 32K (random chosen limit)
+ bail!("too many bytes");
+ }
+ }
+ Ok(())
+ }
+}
+
+impl hyper::service::Service<Uri> for HttpsConnector {
+ type Response = MaybeTlsStream<TcpStream>;
+ type Error = Error;
+ #[allow(clippy::type_complexity)]
+ type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
+
+ fn poll_ready(&mut self, ctx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
+ self.connector
+ .poll_ready(ctx)
+ .map_err(|err| err.into())
+ }
+
+ fn call(&mut self, dst: Uri) -> Self::Future {
+ let mut connector = self.connector.clone();
+ let ssl_connector = Arc::clone(&self.ssl_connector);
+ let is_https = dst.scheme() == Some(&http::uri::Scheme::HTTPS);
+ let host = match dst.host() {
+ Some(host) => host.to_owned(),
+ None => {
+ return futures::future::err(format_err!("missing URL scheme")).boxed();
+ }
+ };
+ let port = dst.port_u16().unwrap_or(if is_https { 443 } else { 80 });
+ let keepalive = self.tcp_keepalive;
+
+ if let Some(ref proxy) = self.proxy {
+
+ let use_connect = is_https || proxy.force_connect;
+
+ let proxy_authority = match helpers::build_authority(&proxy.host, proxy.port) {
+ Ok(authority) => authority,
+ Err(err) => return futures::future::err(err).boxed(),
+ };
+
+ let proxy_uri = match Uri::builder()
+ .scheme("http")
+ .authority(proxy_authority.as_str())
+ .path_and_query("/")
+ .build()
+ {
+ Ok(uri) => uri,
+ Err(err) => return futures::future::err(err.into()).boxed(),
+ };
+
+ let authorization = proxy.authorization.clone();
+
+ if use_connect {
+ async move {
+
+ let mut tcp_stream = connector
+ .call(proxy_uri)
+ .await
+ .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+
+ let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
+ if let Some(authorization) = authorization {
+ connect_request.push_str(&format!("Proxy-Authorization: {}\r\n", authorization));
+ }
+ connect_request.push_str(&format!("Host: {0}:{1}\r\n\r\n", host, port));
+
+ tcp_stream.write_all(connect_request.as_bytes()).await?;
+ tcp_stream.flush().await?;
+
+ Self::parse_connect_response(&mut tcp_stream).await?;
+
+ if is_https {
+ Self::secure_stream(tcp_stream, &ssl_connector, &host).await
+ } else {
+ Ok(MaybeTlsStream::Normal(tcp_stream))
+ }
+ }.boxed()
+ } else {
+ async move {
+ let tcp_stream = connector
+ .call(proxy_uri)
+ .await
+ .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+
+ Ok(MaybeTlsStream::Proxied(tcp_stream))
+ }.boxed()
+ }
+ } else {
+ async move {
+ let dst_str = dst.to_string(); // for error messages
+ let tcp_stream = connector
+ .call(dst)
+ .await
+ .map_err(|err| format_err!("error connecting to {} - {}", dst_str, err))?;
+
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+
+ if is_https {
+ Self::secure_stream(tcp_stream, &ssl_connector, &host).await
+ } else {
+ Ok(MaybeTlsStream::Normal(tcp_stream))
+ }
+ }.boxed()
+ }
+ }
+}
diff --git a/proxmox-http/src/http/mod.rs b/proxmox-http/src/http/mod.rs
index 055648e..3803713 100644
--- a/proxmox-http/src/http/mod.rs
+++ b/proxmox-http/src/http/mod.rs
@@ -5,3 +5,6 @@ pub mod helpers;
mod proxy_config;
pub use proxy_config::ProxyConfig;
+
+#[cfg(feature = "client")]
+pub mod client;
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
index 5e3f9ec..aa44c0d 100644
--- a/proxmox-http/src/lib.rs
+++ b/proxmox-http/src/lib.rs
@@ -1,5 +1,5 @@
#[cfg(feature = "websocket")]
pub mod websocket;
-#[cfg(feature = "http-helpers")]
+#[cfg(any(feature = "http-helpers", feature = "client"))]
pub mod http;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client from proxmox_backup
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (8 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy Fabian Grünbichler
` (10 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
adapted to use already moved helpers/code.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/http/client.rs | 4 +
proxmox-http/src/http/client/simple.rs | 157 +++++++++++++++++++++++++
2 files changed, 161 insertions(+)
create mode 100644 proxmox-http/src/http/client/simple.rs
diff --git a/proxmox-http/src/http/client.rs b/proxmox-http/src/http/client.rs
index 21a65e3..c55cbd4 100644
--- a/proxmox-http/src/http/client.rs
+++ b/proxmox-http/src/http/client.rs
@@ -1,3 +1,7 @@
mod connector;
pub use connector::HttpsConnector;
+
+mod simple;
+pub use simple::SimpleHttp;
+pub use simple::SimpleHttpOptions;
diff --git a/proxmox-http/src/http/client/simple.rs b/proxmox-http/src/http/client/simple.rs
new file mode 100644
index 0000000..110fa55
--- /dev/null
+++ b/proxmox-http/src/http/client/simple.rs
@@ -0,0 +1,157 @@
+use anyhow::{Error, format_err, bail};
+use std::collections::HashMap;
+
+use hyper::Body;
+use hyper::client::{Client, HttpConnector};
+use http::{Request, Response, HeaderValue};
+use openssl::ssl::{SslConnector, SslMethod};
+use futures::*;
+
+use crate::http::{
+ ProxyConfig,
+ client::HttpsConnector,
+};
+
+/// Options for a SimpleHttp client.
+#[derive(Default)]
+pub struct SimpleHttpOptions {
+ /// Proxy configuration
+ pub proxy_config: Option<ProxyConfig>,
+ /// `User-Agent` header value, defaults to `proxmox-simple-http-client/0.1`
+ pub user_agent: Option<String>,
+ /// TCP keepalive time, defaults to 7200
+ pub tcp_keepalive: Option<u32>,
+}
+
+impl SimpleHttpOptions {
+ fn get_proxy_authorization(&self) -> Option<String> {
+ if let Some(ref proxy_config) = self.proxy_config {
+ if !proxy_config.force_connect {
+ return proxy_config.authorization.clone();
+ }
+ }
+
+ None
+ }
+}
+
+/// Asyncrounous HTTP client implementation
+pub struct SimpleHttp {
+ client: Client<HttpsConnector, Body>,
+ options: SimpleHttpOptions,
+}
+
+impl SimpleHttp {
+ pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-simple-http-client/0.1";
+
+ pub fn new() -> Self {
+ Self::with_options(SimpleHttpOptions::default())
+ }
+
+ pub fn with_options(options: SimpleHttpOptions) -> Self {
+ let ssl_connector = SslConnector::builder(SslMethod::tls()).unwrap().build();
+ Self::with_ssl_connector(ssl_connector, options)
+ }
+
+ pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
+ let connector = HttpConnector::new();
+ let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
+ if let Some(ref proxy_config) = options.proxy_config {
+ https.set_proxy(proxy_config.clone());
+ }
+ let client = Client::builder().build(https);
+ Self { client, options }
+ }
+
+ pub fn set_user_agent(&mut self, user_agent: &str) -> Result<(), Error> {
+ self.options.user_agent = Some(user_agent.to_owned());
+ Ok(())
+ }
+
+ fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
+ if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
+ if let Some(ref authorization) = self.options.get_proxy_authorization() {
+ request
+ .headers_mut()
+ .insert(
+ http::header::PROXY_AUTHORIZATION,
+ HeaderValue::from_str(authorization)?,
+ );
+ }
+ }
+ Ok(())
+ }
+
+ pub async fn request(&self, mut request: Request<Body>) -> Result<Response<Body>, Error> {
+ let user_agent = if let Some(ref user_agent) = self.options.user_agent {
+ HeaderValue::from_str(&user_agent)?
+ } else {
+ HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
+ };
+
+ request.headers_mut().insert(hyper::header::USER_AGENT, user_agent);
+
+ self.add_proxy_headers(&mut request)?;
+
+ self.client.request(request)
+ .map_err(Error::from)
+ .await
+ }
+
+ pub async fn post(
+ &mut self,
+ uri: &str,
+ body: Option<String>,
+ content_type: Option<&str>,
+ ) -> Result<Response<Body>, Error> {
+
+ let body = if let Some(body) = body {
+ Body::from(body)
+ } else {
+ Body::empty()
+ };
+ let content_type = content_type.unwrap_or("application/json");
+
+ let request = Request::builder()
+ .method("POST")
+ .uri(uri)
+ .header(hyper::header::CONTENT_TYPE, content_type)
+ .body(body)?;
+
+ self.request(request).await
+ }
+
+ pub async fn get_string(
+ &mut self,
+ uri: &str,
+ extra_headers: Option<&HashMap<String, String>>,
+ ) -> Result<String, Error> {
+
+ let mut request = Request::builder()
+ .method("GET")
+ .uri(uri);
+
+ if let Some(hs) = extra_headers {
+ for (h, v) in hs.iter() {
+ request = request.header(h, v);
+ }
+ }
+
+ let request = request.body(Body::empty())?;
+
+ let res = self.request(request).await?;
+
+ let status = res.status();
+ if !status.is_success() {
+ bail!("Got bad status '{}' from server", status)
+ }
+
+ Self::response_body_string(res).await
+ }
+
+ pub async fn response_body_string(res: Response<Body>) -> Result<String, Error> {
+ let buf = hyper::body::to_bytes(res).await?;
+ String::from_utf8(buf.to_vec())
+ .map_err(|err| format_err!("Error converting HTTP result data: {}", err))
+ }
+}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (9 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 12/13] http: rustfmt Fabian Grünbichler
` (9 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
again.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/http/client/simple.rs | 6 ++++++
proxmox-http/src/http/proxy_config.rs | 2 +-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/proxmox-http/src/http/client/simple.rs b/proxmox-http/src/http/client/simple.rs
index 110fa55..6783781 100644
--- a/proxmox-http/src/http/client/simple.rs
+++ b/proxmox-http/src/http/client/simple.rs
@@ -155,3 +155,9 @@ impl SimpleHttp {
.map_err(|err| format_err!("Error converting HTTP result data: {}", err))
}
}
+
+impl Default for SimpleHttp {
+ fn default() -> Self {
+ Self::new()
+ }
+}
diff --git a/proxmox-http/src/http/proxy_config.rs b/proxmox-http/src/http/proxy_config.rs
index 7e93a47..9eac3bd 100644
--- a/proxmox-http/src/http/proxy_config.rs
+++ b/proxmox-http/src/http/proxy_config.rs
@@ -20,7 +20,7 @@ impl ProxyConfig {
// We only support/use ALL_PROXY environment
match std::env::var_os("ALL_PROXY") {
- None => return Ok(None),
+ None => Ok(None),
Some(all_proxy) => {
let all_proxy = match all_proxy.to_str() {
Some(s) => String::from(s),
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 12/13] http: rustfmt
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (10 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 13/13] http: update d/control Fabian Grünbichler
` (8 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/src/http/client/connector.rs | 78 ++++++++++++-----------
proxmox-http/src/http/client/simple.rs | 45 ++++++-------
proxmox-http/src/http/helpers.rs | 11 ++--
proxmox-http/src/http/proxy_config.rs | 8 +--
proxmox-http/src/http/wrapper.rs | 65 +++++--------------
5 files changed, 89 insertions(+), 118 deletions(-)
diff --git a/proxmox-http/src/http/client/connector.rs b/proxmox-http/src/http/client/connector.rs
index a302dd1..d69fdf2 100644
--- a/proxmox-http/src/http/client/connector.rs
+++ b/proxmox-http/src/http/client/connector.rs
@@ -1,4 +1,4 @@
-use anyhow::{Error, format_err, bail};
+use anyhow::{bail, format_err, Error};
use std::os::unix::io::AsRawFd;
use std::pin::Pin;
use std::sync::Arc;
@@ -8,11 +8,7 @@ use futures::*;
use http::Uri;
use hyper::client::HttpConnector;
use openssl::ssl::SslConnector;
-use tokio::io::{
- AsyncRead,
- AsyncReadExt,
- AsyncWriteExt,
-};
+use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio_openssl::SslStream;
@@ -29,7 +25,11 @@ pub struct HttpsConnector {
}
impl HttpsConnector {
- pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
+ pub fn with_connector(
+ mut connector: HttpConnector,
+ ssl_connector: SslConnector,
+ tcp_keepalive: u32,
+ ) -> Self {
connector.enforce_http(false);
Self {
connector,
@@ -61,21 +61,27 @@ impl HttpsConnector {
Ok(())
}
- async fn parse_connect_response<R: AsyncRead + Unpin>(
- stream: &mut R,
- ) -> Result<(), Error> {
-
+ async fn parse_connect_response<R: AsyncRead + Unpin>(stream: &mut R) -> Result<(), Error> {
let mut data: Vec<u8> = Vec::new();
let mut buffer = [0u8; 256];
const END_MARK: &[u8; 4] = b"\r\n\r\n";
'outer: loop {
let n = stream.read(&mut buffer[..]).await?;
- if n == 0 { break; }
- let search_start = if data.len() > END_MARK.len() { data.len() - END_MARK.len() + 1 } else { 0 };
+ if n == 0 {
+ break;
+ }
+ let search_start = if data.len() > END_MARK.len() {
+ data.len() - END_MARK.len() + 1
+ } else {
+ 0
+ };
data.extend(&buffer[..n]);
if data.len() >= END_MARK.len() {
- if let Some(pos) = data[search_start..].windows(END_MARK.len()).position(|w| w == END_MARK) {
+ if let Some(pos) = data[search_start..]
+ .windows(END_MARK.len())
+ .position(|w| w == END_MARK)
+ {
let response = String::from_utf8_lossy(&data);
let status_line = match response.split("\r\n").next() {
Some(status) => status,
@@ -89,7 +95,8 @@ impl HttpsConnector {
break 'outer;
}
}
- if data.len() > 1024*32 { // max 32K (random chosen limit)
+ if data.len() > 1024 * 32 {
+ // max 32K (random chosen limit)
bail!("too many bytes");
}
}
@@ -101,12 +108,11 @@ impl hyper::service::Service<Uri> for HttpsConnector {
type Response = MaybeTlsStream<TcpStream>;
type Error = Error;
#[allow(clippy::type_complexity)]
- type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
+ type Future =
+ Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
fn poll_ready(&mut self, ctx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
- self.connector
- .poll_ready(ctx)
- .map_err(|err| err.into())
+ self.connector.poll_ready(ctx).map_err(|err| err.into())
}
fn call(&mut self, dst: Uri) -> Self::Future {
@@ -123,7 +129,6 @@ impl hyper::service::Service<Uri> for HttpsConnector {
let keepalive = self.tcp_keepalive;
if let Some(ref proxy) = self.proxy {
-
let use_connect = is_https || proxy.force_connect;
let proxy_authority = match helpers::build_authority(&proxy.host, proxy.port) {
@@ -145,17 +150,16 @@ impl hyper::service::Service<Uri> for HttpsConnector {
if use_connect {
async move {
-
- let mut tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+ let mut tcp_stream = connector.call(proxy_uri).await.map_err(|err| {
+ format_err!("error connecting to {} - {}", proxy_authority, err)
+ })?;
let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
if let Some(authorization) = authorization {
- connect_request.push_str(&format!("Proxy-Authorization: {}\r\n", authorization));
+ connect_request
+ .push_str(&format!("Proxy-Authorization: {}\r\n", authorization));
}
connect_request.push_str(&format!("Host: {0}:{1}\r\n\r\n", host, port));
@@ -169,18 +173,19 @@ impl hyper::service::Service<Uri> for HttpsConnector {
} else {
Ok(MaybeTlsStream::Normal(tcp_stream))
}
- }.boxed()
+ }
+ .boxed()
} else {
- async move {
- let tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
+ async move {
+ let tcp_stream = connector.call(proxy_uri).await.map_err(|err| {
+ format_err!("error connecting to {} - {}", proxy_authority, err)
+ })?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
- Ok(MaybeTlsStream::Proxied(tcp_stream))
- }.boxed()
+ Ok(MaybeTlsStream::Proxied(tcp_stream))
+ }
+ .boxed()
}
} else {
async move {
@@ -197,7 +202,8 @@ impl hyper::service::Service<Uri> for HttpsConnector {
} else {
Ok(MaybeTlsStream::Normal(tcp_stream))
}
- }.boxed()
+ }
+ .boxed()
}
}
}
diff --git a/proxmox-http/src/http/client/simple.rs b/proxmox-http/src/http/client/simple.rs
index 6783781..b6e6d5a 100644
--- a/proxmox-http/src/http/client/simple.rs
+++ b/proxmox-http/src/http/client/simple.rs
@@ -1,16 +1,13 @@
-use anyhow::{Error, format_err, bail};
+use anyhow::{bail, format_err, Error};
use std::collections::HashMap;
-use hyper::Body;
+use futures::*;
+use http::{HeaderValue, Request, Response};
use hyper::client::{Client, HttpConnector};
-use http::{Request, Response, HeaderValue};
+use hyper::Body;
use openssl::ssl::{SslConnector, SslMethod};
-use futures::*;
-use crate::http::{
- ProxyConfig,
- client::HttpsConnector,
-};
+use crate::http::{client::HttpsConnector, ProxyConfig};
/// Options for a SimpleHttp client.
#[derive(Default)]
@@ -27,7 +24,7 @@ impl SimpleHttpOptions {
fn get_proxy_authorization(&self) -> Option<String> {
if let Some(ref proxy_config) = self.proxy_config {
if !proxy_config.force_connect {
- return proxy_config.authorization.clone();
+ return proxy_config.authorization.clone();
}
}
@@ -55,7 +52,11 @@ impl SimpleHttp {
pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
+ let mut https = HttpsConnector::with_connector(
+ connector,
+ ssl_connector,
+ options.tcp_keepalive.unwrap_or(7200),
+ );
if let Some(ref proxy_config) = options.proxy_config {
https.set_proxy(proxy_config.clone());
}
@@ -71,12 +72,10 @@ impl SimpleHttp {
fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
if let Some(ref authorization) = self.options.get_proxy_authorization() {
- request
- .headers_mut()
- .insert(
- http::header::PROXY_AUTHORIZATION,
- HeaderValue::from_str(authorization)?,
- );
+ request.headers_mut().insert(
+ http::header::PROXY_AUTHORIZATION,
+ HeaderValue::from_str(authorization)?,
+ );
}
}
Ok(())
@@ -89,13 +88,13 @@ impl SimpleHttp {
HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
};
- request.headers_mut().insert(hyper::header::USER_AGENT, user_agent);
+ request
+ .headers_mut()
+ .insert(hyper::header::USER_AGENT, user_agent);
self.add_proxy_headers(&mut request)?;
- self.client.request(request)
- .map_err(Error::from)
- .await
+ self.client.request(request).map_err(Error::from).await
}
pub async fn post(
@@ -104,7 +103,6 @@ impl SimpleHttp {
body: Option<String>,
content_type: Option<&str>,
) -> Result<Response<Body>, Error> {
-
let body = if let Some(body) = body {
Body::from(body)
} else {
@@ -126,10 +124,7 @@ impl SimpleHttp {
uri: &str,
extra_headers: Option<&HashMap<String, String>>,
) -> Result<String, Error> {
-
- let mut request = Request::builder()
- .method("GET")
- .uri(uri);
+ let mut request = Request::builder().method("GET").uri(uri);
if let Some(hs) = extra_headers {
for (h, v) in hs.iter() {
diff --git a/proxmox-http/src/http/helpers.rs b/proxmox-http/src/http/helpers.rs
index 3f663d2..ca67161 100644
--- a/proxmox-http/src/http/helpers.rs
+++ b/proxmox-http/src/http/helpers.rs
@@ -6,10 +6,11 @@ use http::uri::Authority;
pub fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
let bytes = host.as_bytes();
let len = bytes.len();
- let authority = if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len-1] != b']' {
- format!("[{}]:{}", host, port).parse()?
- } else {
- format!("{}:{}", host, port).parse()?
- };
+ let authority =
+ if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len - 1] != b']' {
+ format!("[{}]:{}", host, port).parse()?
+ } else {
+ format!("{}:{}", host, port).parse()?
+ };
Ok(authority)
}
diff --git a/proxmox-http/src/http/proxy_config.rs b/proxmox-http/src/http/proxy_config.rs
index 9eac3bd..3fe7cba 100644
--- a/proxmox-http/src/http/proxy_config.rs
+++ b/proxmox-http/src/http/proxy_config.rs
@@ -1,4 +1,4 @@
-use anyhow::{Error, format_err, bail};
+use anyhow::{bail, format_err, Error};
use http::Uri;
@@ -16,7 +16,6 @@ pub struct ProxyConfig {
impl ProxyConfig {
/// Parse proxy config from ALL_PROXY environment var
pub fn from_proxy_env() -> Result<Option<ProxyConfig>, Error> {
-
// We only support/use ALL_PROXY environment
match std::env::var_os("ALL_PROXY") {
@@ -70,7 +69,8 @@ impl ProxyConfig {
authorization,
force_connect: false,
})
- }).map_err(|err| format_err!("parse_proxy_url failed: {}", err))
+ })
+ .map_err(|err| format_err!("parse_proxy_url failed: {}", err))
}
/// Assemble canonical proxy string (including scheme and port)
@@ -78,7 +78,7 @@ impl ProxyConfig {
let authority = helpers::build_authority(&self.host, self.port)?;
Ok(match self.authorization {
None => format!("http://{}", authority),
- Some(ref authorization) => format!("http://{}@{}", authorization, authority)
+ Some(ref authorization) => format!("http://{}@{}", authorization, authority),
})
}
}
diff --git a/proxmox-http/src/http/wrapper.rs b/proxmox-http/src/http/wrapper.rs
index 3399b28..9ed4221 100644
--- a/proxmox-http/src/http/wrapper.rs
+++ b/proxmox-http/src/http/wrapper.rs
@@ -2,7 +2,7 @@ use std::io;
use std::pin::Pin;
use std::task::{Context, Poll};
-use hyper::client::connect::{Connection, Connected};
+use hyper::client::connect::{Connected, Connection};
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
use tokio_openssl::SslStream;
@@ -22,15 +22,9 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
buf: &mut ReadBuf,
) -> Poll<Result<(), io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_read(cx, buf),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_read(cx, buf),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_read(cx, buf),
}
}
}
@@ -42,15 +36,9 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
buf: &[u8],
) -> Poll<Result<usize, io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_write(cx, buf),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_write(cx, buf),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_write(cx, buf),
}
}
@@ -60,15 +48,9 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
bufs: &[io::IoSlice<'_>],
) -> Poll<Result<usize, io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_write_vectored(cx, bufs),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_write_vectored(cx, bufs),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_write_vectored(cx, bufs),
}
}
@@ -82,36 +64,23 @@ impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_flush(cx),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_flush(cx),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_flush(cx),
}
}
fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
+ MaybeTlsStream::Normal(ref mut s) => Pin::new(s).poll_shutdown(cx),
+ MaybeTlsStream::Proxied(ref mut s) => Pin::new(s).poll_shutdown(cx),
+ MaybeTlsStream::Secured(ref mut s) => Pin::new(s).poll_shutdown(cx),
}
}
}
// we need this for the hyper http client
-impl <S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S>
-{
+impl<S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S> {
fn connected(&self) -> Connected {
match self {
MaybeTlsStream::Normal(s) => s.connected(),
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox 13/13] http: update d/control
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (11 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 12/13] http: rustfmt Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
` (7 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
proxmox-http/debian/control | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/proxmox-http/debian/control b/proxmox-http/debian/control
index f010d6e..860e705 100644
--- a/proxmox-http/debian/control
+++ b/proxmox-http/debian/control
@@ -118,18 +118,12 @@ Depends:
librust-tokio-1+io-util-dev,
librust-tokio-openssl-0.6+default-dev (>= 0.6.1-~~)
Provides:
- librust-proxmox-http+simple-client-dev (= ${binary:Version}),
librust-proxmox-http-0+http-helpers-dev (= ${binary:Version}),
- librust-proxmox-http-0+simple-client-dev (= ${binary:Version}),
librust-proxmox-http-0.1+http-helpers-dev (= ${binary:Version}),
- librust-proxmox-http-0.1+simple-client-dev (= ${binary:Version}),
- librust-proxmox-http-0.1.0+http-helpers-dev (= ${binary:Version}),
- librust-proxmox-http-0.1.0+simple-client-dev (= ${binary:Version})
-Description: Proxmox HTTP library - feature "http-helpers" and 1 more
+ librust-proxmox-http-0.1.0+http-helpers-dev (= ${binary:Version})
+Description: Proxmox HTTP library - feature "http-helpers"
This metapackage enables feature "http-helpers" for the Rust proxmox-http
crate, by pulling in any additional dependencies needed by that feature.
- .
- Additionally, this package also provides the "simple-client" feature.
Package: librust-proxmox-http+hyper-dev
Architecture: any
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (12 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 13/13] http: update d/control Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-17 8:35 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate Fabian Grünbichler
` (6 subsequent siblings)
20 siblings, 1 reply; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires patch #3 in proxmox
Cargo.toml | 5 +++--
debian/control | 3 ++-
src/api2/node.rs | 2 +-
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index cd99241e..997017c0 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -53,10 +53,11 @@ percent-encoding = "2.1"
pin-utils = "0.1.0"
pin-project = "1.0"
pathpatterns = "0.1.2"
-proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro", "websocket" ] }
+proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
-#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro", "websocket" ] }
+#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
proxmox-fuse = "0.1.1"
+proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "websocket" ] }
pxar = { version = "0.10.1", features = [ "tokio-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io" ] }
regex = "1.2"
diff --git a/debian/control b/debian/control
index 3f1edba8..a02518fe 100644
--- a/debian/control
+++ b/debian/control
@@ -42,9 +42,10 @@ Build-Depends: debhelper (>= 11),
librust-proxmox-0.11+api-macro-dev (>= 0.11.4-~~),
librust-proxmox-0.11+default-dev (>= 0.11.4-~~),
librust-proxmox-0.11+sortable-macro-dev (>= 0.11.4-~~),
- librust-proxmox-0.11+websocket-dev (>= 0.11.4-~~),
librust-proxmox-acme-rs-0.2+default-dev (>= 0.2.1-~~),
librust-proxmox-fuse-0.1+default-dev (>= 0.1.1-~~),
+ librust-proxmox-http-0.1+default-dev,
+ librust-proxmox-http-0.1+websocket-dev,
librust-pxar-0.10+default-dev (>= 0.10.1-~~),
librust-pxar-0.10+tokio-io-dev (>= 0.10.1-~~),
librust-regex-1+default-dev (>= 1.2-~~),
diff --git a/src/api2/node.rs b/src/api2/node.rs
index 75271cd5..af9ecf69 100644
--- a/src/api2/node.rs
+++ b/src/api2/node.rs
@@ -17,7 +17,7 @@ use proxmox::api::{
api, schema::*, ApiHandler, ApiMethod, ApiResponseFuture, Permission, RpcEnvironment,
};
use proxmox::list_subdirs_api_method;
-use proxmox::tools::websocket::WebSocket;
+use proxmox_http::websocket::WebSocket;
use proxmox::{identity, sortable};
use crate::api2::types::*;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (13 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http Fabian Grünbichler
` (5 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
and constant to tools module.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #5 and proxmox version bump
src/bin/proxmox-backup-proxy.rs | 6 ++----
src/tools.rs | 4 +++-
src/tools/http.rs | 7 +++----
src/tools/socket.rs | 23 -----------------------
4 files changed, 8 insertions(+), 32 deletions(-)
delete mode 100644 src/tools/socket.rs
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 71517023..a53f554a 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -11,6 +11,7 @@ use serde_json::Value;
use proxmox::try_block;
use proxmox::api::RpcEnvironmentType;
+use proxmox::sys::linux::socket::set_tcp_keepalive;
use proxmox_backup::{
backup::DataStore,
@@ -38,6 +39,7 @@ use proxmox_backup::buildcfg;
use proxmox_backup::server;
use proxmox_backup::auth_helpers::*;
use proxmox_backup::tools::{
+ PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
daemon,
disks::{
DiskManage,
@@ -45,10 +47,6 @@ use proxmox_backup::tools::{
get_pool_from_dataset,
},
logrotate::LogRotate,
- socket::{
- set_tcp_keepalive,
- PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- },
};
use proxmox_backup::api2::pull::do_sync_job;
diff --git a/src/tools.rs b/src/tools.rs
index 8f2acded..8a1d0bc7 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -43,7 +43,6 @@ pub mod lru_cache;
pub mod nom;
pub mod runtime;
pub mod serde_filter;
-pub mod socket;
pub mod statistics;
pub mod subscription;
pub mod systemd;
@@ -483,6 +482,9 @@ impl<T: Any> AsAny for T {
}
}
+/// The default 2 hours are far too long for PBS
+pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
+
/// This used to be: `SIMPLE_ENCODE_SET` plus space, `"`, `#`, `<`, `>`, backtick, `?`, `{`, `}`
pub const DEFAULT_ENCODE_SET: &AsciiSet = &percent_encoding::CONTROLS // 0..1f and 7e
// The SIMPLE_ENCODE_SET adds space and anything >= 0x7e (7e itself is already included above)
diff --git a/src/tools/http.rs b/src/tools/http.rs
index 0821992a..1d96c70f 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -18,12 +18,11 @@ use tokio::{
};
use tokio_openssl::SslStream;
+use proxmox::sys::linux::socket::set_tcp_keepalive;
+
use crate::tools::{
+ PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
async_io::MaybeTlsStream,
- socket::{
- set_tcp_keepalive,
- PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- },
};
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
diff --git a/src/tools/socket.rs b/src/tools/socket.rs
deleted file mode 100644
index 01e5edec..00000000
--- a/src/tools/socket.rs
+++ /dev/null
@@ -1,23 +0,0 @@
-use std::os::unix::io::RawFd;
-
-use nix::sys::socket::sockopt::{KeepAlive, TcpKeepIdle};
-use nix::sys::socket::setsockopt;
-
-pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
-
-/// Set TCP keepalive time on a socket
-///
-/// See "man 7 tcp" for details.
-///
-/// The default on Linux is 7200 (2 hours) which is far too long for
-/// our backup tools.
-pub fn set_tcp_keepalive(
- socket_fd: RawFd,
- tcp_keepalive_time: u32,
-) -> nix::Result<()> {
-
- setsockopt(socket_fd, KeepAlive, &true)?;
- setsockopt(socket_fd, TcpKeepIdle, &tcp_keepalive_time)?;
-
- Ok(())
-}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (14 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable Fabian Grünbichler
` (4 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #6
Cargo.toml | 2 +-
src/tools/async_io.rs | 119 +-----------------------------------------
src/tools/http.rs | 6 +--
3 files changed, 4 insertions(+), 123 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index 997017c0..df649c1b 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -57,7 +57,7 @@ proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
proxmox-fuse = "0.1.1"
-proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "websocket" ] }
+proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "http-helpers", "websocket" ] }
pxar = { version = "0.10.1", features = [ "tokio-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io" ] }
regex = "1.2"
diff --git a/src/tools/async_io.rs b/src/tools/async_io.rs
index 83110912..66d38094 100644
--- a/src/tools/async_io.rs
+++ b/src/tools/async_io.rs
@@ -1,131 +1,14 @@
//! AsyncRead/AsyncWrite utilities.
-use std::io;
use std::os::unix::io::{AsRawFd, RawFd};
use std::pin::Pin;
use std::task::{Context, Poll};
use futures::stream::{Stream, TryStream};
use futures::ready;
-use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
+use tokio::io::{AsyncRead, AsyncWrite};
use tokio::net::TcpListener;
-use tokio_openssl::SslStream;
-use hyper::client::connect::{Connection, Connected};
-/// Asynchronous stream, possibly encrypted and proxied
-///
-/// Usefule for HTTP client implementations using hyper.
-pub enum MaybeTlsStream<S> {
- Normal(S),
- Proxied(S),
- Secured(SslStream<S>),
-}
-
-impl<S: AsyncRead + AsyncWrite + Unpin> AsyncRead for MaybeTlsStream<S> {
- fn poll_read(
- self: Pin<&mut Self>,
- cx: &mut Context,
- buf: &mut ReadBuf,
- ) -> Poll<Result<(), io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_read(cx, buf)
- }
- }
- }
-}
-
-impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MaybeTlsStream<S> {
- fn poll_write(
- self: Pin<&mut Self>,
- cx: &mut Context,
- buf: &[u8],
- ) -> Poll<Result<usize, io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write(cx, buf)
- }
- }
- }
-
- fn poll_write_vectored(
- self: Pin<&mut Self>,
- cx: &mut Context<'_>,
- bufs: &[io::IoSlice<'_>],
- ) -> Poll<Result<usize, io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_write_vectored(cx, bufs)
- }
- }
- }
-
- fn is_write_vectored(&self) -> bool {
- match self {
- MaybeTlsStream::Normal(s) => s.is_write_vectored(),
- MaybeTlsStream::Proxied(s) => s.is_write_vectored(),
- MaybeTlsStream::Secured(s) => s.is_write_vectored(),
- }
- }
-
- fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_flush(cx)
- }
- }
- }
-
- fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Result<(), io::Error>> {
- match self.get_mut() {
- MaybeTlsStream::Normal(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Proxied(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- MaybeTlsStream::Secured(ref mut s) => {
- Pin::new(s).poll_shutdown(cx)
- }
- }
- }
-}
-
-// we need this for the hyper http client
-impl <S: Connection + AsyncRead + AsyncWrite + Unpin> Connection for MaybeTlsStream<S>
-{
- fn connected(&self) -> Connected {
- match self {
- MaybeTlsStream::Normal(s) => s.connected(),
- MaybeTlsStream::Proxied(s) => s.connected().proxy(true),
- MaybeTlsStream::Secured(s) => s.get_ref().connected(),
- }
- }
-}
/// Tokio's `Incoming` now is a reference type and hyper's `AddrIncoming` misses some standard
/// stuff like `AsRawFd`, so here's something implementing hyper's `Accept` from a `TcpListener`
diff --git a/src/tools/http.rs b/src/tools/http.rs
index 1d96c70f..0f5b8470 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -19,11 +19,9 @@ use tokio::{
use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
+use proxmox_http::http::MaybeTlsStream;
-use crate::tools::{
- PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- async_io::MaybeTlsStream,
-};
+use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (15 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http Fabian Grünbichler
` (3 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
it's the only PBS-specific part in there, so let's make it
product-agnostic before moving it off to proxmox-http.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
src/client/http_client.rs | 3 ++-
src/tools/http.rs | 13 +++++++------
src/tools/simple_http_client.rs | 3 ++-
3 files changed, 11 insertions(+), 8 deletions(-)
diff --git a/src/client/http_client.rs b/src/client/http_client.rs
index 7fe33bcc..056f30e5 100644
--- a/src/client/http_client.rs
+++ b/src/client/http_client.rs
@@ -26,6 +26,7 @@ use crate::tools::{
self,
BroadcastFuture,
DEFAULT_ENCODE_SET,
+ PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
http::{
build_authority,
HttpsConnector,
@@ -343,7 +344,7 @@ impl HttpClient {
httpc.enforce_http(false); // we want https...
httpc.set_connect_timeout(Some(std::time::Duration::new(10, 0)));
- let https = HttpsConnector::with_connector(httpc, ssl_connector_builder.build());
+ let https = HttpsConnector::with_connector(httpc, ssl_connector_builder.build(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
let client = Client::builder()
//.http2_initial_stream_window_size( (1 << 31) - 2)
diff --git a/src/tools/http.rs b/src/tools/http.rs
index 0f5b8470..a6b92aad 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -21,8 +21,6 @@ use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
use proxmox_http::http::MaybeTlsStream;
-use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
-
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
let bytes = host.as_bytes();
@@ -120,15 +118,17 @@ pub struct HttpsConnector {
connector: HttpConnector,
ssl_connector: Arc<SslConnector>,
proxy: Option<ProxyConfig>,
+ tcp_keepalive: u32,
}
impl HttpsConnector {
- pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector) -> Self {
+ pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
connector.enforce_http(false);
Self {
connector,
ssl_connector: Arc::new(ssl_connector),
proxy: None,
+ tcp_keepalive,
}
}
@@ -213,6 +213,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
}
};
let port = dst.port_u16().unwrap_or(if is_https { 443 } else { 80 });
+ let keepalive = self.tcp_keepalive;
if let Some(ref proxy) = self.proxy {
@@ -243,7 +244,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
.await
.map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
if let Some(authorization) = authorization {
@@ -272,7 +273,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
.await
.map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
Ok(MaybeTlsStream::Proxied(tcp_stream))
}.boxed()
@@ -285,7 +286,7 @@ impl hyper::service::Service<Uri> for HttpsConnector {
.await
.map_err(|err| format_err!("error connecting to {} - {}", dst_str, err))?;
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
+ let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
if is_https {
Self::secure_stream(tcp_stream, &ssl_connector, &host).await
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index ca11ded8..1e399267 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -7,6 +7,7 @@ use http::{Request, Response, HeaderValue};
use openssl::ssl::{SslConnector, SslMethod};
use futures::*;
+use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
use crate::tools::http::{HttpsConnector, ProxyConfig};
/// Asyncrounous HTTP client implementation
@@ -35,7 +36,7 @@ impl SimpleHttp {
}
let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector);
+ let mut https = HttpsConnector::with_connector(connector, ssl_connector, PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
if let Some(proxy_config) = proxy_config {
https.set_proxy(proxy_config);
}
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (16 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http " Fabian Grünbichler
` (2 subsequent siblings)
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #7 & #8
src/api2/node/apt.rs | 4 +-
src/api2/types/mod.rs | 2 +-
src/config/node.rs | 3 +-
src/tools/http.rs | 82 +--------------------------------
src/tools/simple_http_client.rs | 4 +-
5 files changed, 10 insertions(+), 85 deletions(-)
diff --git a/src/api2/node/apt.rs b/src/api2/node/apt.rs
index c149a14b..120d5339 100644
--- a/src/api2/node/apt.rs
+++ b/src/api2/node/apt.rs
@@ -7,9 +7,11 @@ use proxmox::api::{api, RpcEnvironment, RpcEnvironmentType, Permission};
use proxmox::api::router::{Router, SubdirMap};
use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_http::http::ProxyConfig;
+
use crate::config::node;
use crate::server::WorkerTask;
-use crate::tools::{apt, SimpleHttp, http::ProxyConfig, subscription};
+use crate::tools::{apt, SimpleHttp, subscription};
use crate::config::acl::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY};
use crate::api2::types::{Authid, APTUpdateInfo, NODE_SCHEMA, UPID_SCHEMA};
diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index e42083f0..26b99790 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -1660,7 +1660,7 @@ pub struct NodeStatus {
pub const HTTP_PROXY_SCHEMA: Schema = StringSchema::new(
"HTTP proxy configuration [http://]<host>[:port]")
.format(&ApiStringFormat::VerifyFn(|s| {
- crate::tools::http::ProxyConfig::parse_proxy_url(s)?;
+ proxmox_http::http::ProxyConfig::parse_proxy_url(s)?;
Ok(())
}))
.min_length(1)
diff --git a/src/config/node.rs b/src/config/node.rs
index e818e47e..b003ae01 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -10,11 +10,12 @@ use proxmox::api::api;
use proxmox::api::schema::{ApiStringFormat, Updater};
use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_http::http::ProxyConfig;
+
use crate::acme::AcmeClient;
use crate::api2::types::{
AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
};
-use crate::tools::http::ProxyConfig;
const CONF_FILE: &str = configdir!("/node.cfg");
const LOCK_FILE: &str = configdir!("/.node.lck");
diff --git a/src/tools/http.rs b/src/tools/http.rs
index a6b92aad..b99d26a1 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -19,7 +19,7 @@ use tokio::{
use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
-use proxmox_http::http::MaybeTlsStream;
+use proxmox_http::http::{MaybeTlsStream, ProxyConfig};
// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
@@ -33,86 +33,6 @@ pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error>
Ok(authority)
}
-/// HTTP Proxy Configuration
-#[derive(Clone)]
-pub struct ProxyConfig {
- pub host: String,
- pub port: u16,
- pub authorization: Option<String>, // user:pass
- pub force_connect: bool,
-}
-
-impl ProxyConfig {
-
- /// Parse proxy config from ALL_PROXY environment var
- pub fn from_proxy_env() -> Result<Option<ProxyConfig>, Error> {
-
- // We only support/use ALL_PROXY environment
-
- match std::env::var_os("ALL_PROXY") {
- None => return Ok(None),
- Some(all_proxy) => {
- let all_proxy = match all_proxy.to_str() {
- Some(s) => String::from(s),
- None => bail!("non UTF-8 content in env ALL_PROXY"),
- };
- if all_proxy.is_empty() {
- return Ok(None);
- }
- let config = Self::parse_proxy_url(&all_proxy)?;
- Ok(Some(config))
- }
- }
- }
-
- /// Parse proxy configuration string [http://]<host>[:port]
- ///
- /// Default port is 1080 (like curl)
- pub fn parse_proxy_url(http_proxy: &str) -> Result<ProxyConfig, Error> {
- proxmox::try_block!({
- let proxy_uri: Uri = http_proxy.parse()?;
- let proxy_authority = match proxy_uri.authority() {
- Some(authority) => authority,
- None => bail!("missing proxy authority"),
- };
- let host = proxy_authority.host().to_owned();
- let port = match proxy_uri.port() {
- Some(port) => port.as_u16(),
- None => 1080, // CURL default port
- };
-
- match proxy_uri.scheme_str() {
- Some("http") => { /* Ok */ }
- Some(scheme) => bail!("unsupported proxy scheme '{}'", scheme),
- None => { /* assume HTTP */ }
- }
-
- let authority_vec: Vec<&str> = proxy_authority.as_str().rsplitn(2, '@').collect();
- let authorization = if authority_vec.len() == 2 {
- Some(authority_vec[1].to_string())
- } else {
- None
- };
-
- Ok(ProxyConfig {
- host,
- port,
- authorization,
- force_connect: false,
- })
- }).map_err(|err| format_err!("parse_proxy_url failed: {}", err))
- }
-
- /// Assemble canonical proxy string (including scheme and port)
- pub fn to_proxy_string(&self) -> Result<String, Error> {
- let authority = build_authority(&self.host, self.port)?;
- Ok(match self.authorization {
- None => format!("http://{}", authority),
- Some(ref authorization) => format!("http://{}@{}", authorization, authority)
- })
- }
-}
-
#[derive(Clone)]
pub struct HttpsConnector {
connector: HttpConnector,
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index 1e399267..729711c8 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -7,8 +7,10 @@ use http::{Request, Response, HeaderValue};
use openssl::ssl::{SslConnector, SslMethod};
use futures::*;
+use proxmox_http::http::ProxyConfig;
+
use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
-use crate::tools::http::{HttpsConnector, ProxyConfig};
+use crate::tools::http::HttpsConnector;
/// Asyncrounous HTTP client implementation
pub struct SimpleHttp {
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (17 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http Fabian Grünbichler
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #7-9
Cargo.toml | 2 +-
src/client/http_client.rs | 7 +-
src/tools.rs | 1 -
src/tools/http.rs | 197 --------------------------------
src/tools/simple_http_client.rs | 6 +-
5 files changed, 8 insertions(+), 205 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index df649c1b..73bd4240 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -57,7 +57,7 @@ proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
proxmox-fuse = "0.1.1"
-proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "http-helpers", "websocket" ] }
+proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "client", "http-helpers", "websocket" ] }
pxar = { version = "0.10.1", features = [ "tokio-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io" ] }
regex = "1.2"
diff --git a/src/client/http_client.rs b/src/client/http_client.rs
index 056f30e5..4c548481 100644
--- a/src/client/http_client.rs
+++ b/src/client/http_client.rs
@@ -20,6 +20,9 @@ use proxmox::{
tools::fs::{file_get_json, replace_file, CreateOptions},
};
+use proxmox_http::http::client::HttpsConnector;
+use proxmox_http::http::helpers::build_authority;
+
use super::pipe_to_stream::PipeToSendStream;
use crate::api2::types::{Authid, Userid};
use crate::tools::{
@@ -27,10 +30,6 @@ use crate::tools::{
BroadcastFuture,
DEFAULT_ENCODE_SET,
PROXMOX_BACKUP_TCP_KEEPALIVE_TIME,
- http::{
- build_authority,
- HttpsConnector,
- },
};
/// Timeout used for several HTTP operations that are expected to finish quickly but may block in
diff --git a/src/tools.rs b/src/tools.rs
index 8a1d0bc7..eb9a9fd0 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -31,7 +31,6 @@ pub mod disks;
pub mod format;
pub mod fs;
pub mod fuse_loop;
-pub mod http;
mod simple_http_client;
pub use simple_http_client::SimpleHttp;
diff --git a/src/tools/http.rs b/src/tools/http.rs
index b99d26a1..dba8dc98 100644
--- a/src/tools/http.rs
+++ b/src/tools/http.rs
@@ -20,200 +20,3 @@ use tokio_openssl::SslStream;
use proxmox::sys::linux::socket::set_tcp_keepalive;
use proxmox_http::http::{MaybeTlsStream, ProxyConfig};
-
-// Build a http::uri::Authority ("host:port"), use '[..]' around IPv6 addresses
-pub(crate) fn build_authority(host: &str, port: u16) -> Result<Authority, Error> {
- let bytes = host.as_bytes();
- let len = bytes.len();
- let authority = if len > 3 && bytes.contains(&b':') && bytes[0] != b'[' && bytes[len-1] != b']' {
- format!("[{}]:{}", host, port).parse()?
- } else {
- format!("{}:{}", host, port).parse()?
- };
- Ok(authority)
-}
-
-#[derive(Clone)]
-pub struct HttpsConnector {
- connector: HttpConnector,
- ssl_connector: Arc<SslConnector>,
- proxy: Option<ProxyConfig>,
- tcp_keepalive: u32,
-}
-
-impl HttpsConnector {
- pub fn with_connector(mut connector: HttpConnector, ssl_connector: SslConnector, tcp_keepalive: u32) -> Self {
- connector.enforce_http(false);
- Self {
- connector,
- ssl_connector: Arc::new(ssl_connector),
- proxy: None,
- tcp_keepalive,
- }
- }
-
- pub fn set_proxy(&mut self, proxy: ProxyConfig) {
- self.proxy = Some(proxy);
- }
-
- async fn secure_stream(
- tcp_stream: TcpStream,
- ssl_connector: &SslConnector,
- host: &str,
- ) -> Result<MaybeTlsStream<TcpStream>, Error> {
- let config = ssl_connector.configure()?;
- let mut conn: SslStream<TcpStream> = SslStream::new(config.into_ssl(host)?, tcp_stream)?;
- Pin::new(&mut conn).connect().await?;
- Ok(MaybeTlsStream::Secured(conn))
- }
-
- fn parse_status_line(status_line: &str) -> Result<(), Error> {
- if !(status_line.starts_with("HTTP/1.1 200") || status_line.starts_with("HTTP/1.0 200")) {
- bail!("proxy connect failed - invalid status: {}", status_line)
- }
- Ok(())
- }
-
- async fn parse_connect_response<R: AsyncRead + Unpin>(
- stream: &mut R,
- ) -> Result<(), Error> {
-
- let mut data: Vec<u8> = Vec::new();
- let mut buffer = [0u8; 256];
- const END_MARK: &[u8; 4] = b"\r\n\r\n";
-
- 'outer: loop {
- let n = stream.read(&mut buffer[..]).await?;
- if n == 0 { break; }
- let search_start = if data.len() > END_MARK.len() { data.len() - END_MARK.len() + 1 } else { 0 };
- data.extend(&buffer[..n]);
- if data.len() >= END_MARK.len() {
- if let Some(pos) = data[search_start..].windows(END_MARK.len()).position(|w| w == END_MARK) {
- let response = String::from_utf8_lossy(&data);
- let status_line = match response.split("\r\n").next() {
- Some(status) => status,
- None => bail!("missing newline"),
- };
- Self::parse_status_line(status_line)?;
-
- if pos != data.len() - END_MARK.len() {
- bail!("unexpected data after connect response");
- }
- break 'outer;
- }
- }
- if data.len() > 1024*32 { // max 32K (random chosen limit)
- bail!("too many bytes");
- }
- }
- Ok(())
- }
-}
-
-impl hyper::service::Service<Uri> for HttpsConnector {
- type Response = MaybeTlsStream<TcpStream>;
- type Error = Error;
- #[allow(clippy::type_complexity)]
- type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
-
- fn poll_ready(&mut self, ctx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
- self.connector
- .poll_ready(ctx)
- .map_err(|err| err.into())
- }
-
- fn call(&mut self, dst: Uri) -> Self::Future {
- let mut connector = self.connector.clone();
- let ssl_connector = Arc::clone(&self.ssl_connector);
- let is_https = dst.scheme() == Some(&http::uri::Scheme::HTTPS);
- let host = match dst.host() {
- Some(host) => host.to_owned(),
- None => {
- return futures::future::err(format_err!("missing URL scheme")).boxed();
- }
- };
- let port = dst.port_u16().unwrap_or(if is_https { 443 } else { 80 });
- let keepalive = self.tcp_keepalive;
-
- if let Some(ref proxy) = self.proxy {
-
- let use_connect = is_https || proxy.force_connect;
-
- let proxy_authority = match build_authority(&proxy.host, proxy.port) {
- Ok(authority) => authority,
- Err(err) => return futures::future::err(err).boxed(),
- };
-
- let proxy_uri = match Uri::builder()
- .scheme("http")
- .authority(proxy_authority.as_str())
- .path_and_query("/")
- .build()
- {
- Ok(uri) => uri,
- Err(err) => return futures::future::err(err.into()).boxed(),
- };
-
- let authorization = proxy.authorization.clone();
-
- if use_connect {
- async move {
-
- let mut tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
-
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
-
- let mut connect_request = format!("CONNECT {0}:{1} HTTP/1.1\r\n", host, port);
- if let Some(authorization) = authorization {
- connect_request.push_str(&format!(
- "Proxy-Authorization: Basic {}\r\n",
- base64::encode(authorization),
- ));
- }
- connect_request.push_str(&format!("Host: {0}:{1}\r\n\r\n", host, port));
-
- tcp_stream.write_all(connect_request.as_bytes()).await?;
- tcp_stream.flush().await?;
-
- Self::parse_connect_response(&mut tcp_stream).await?;
-
- if is_https {
- Self::secure_stream(tcp_stream, &ssl_connector, &host).await
- } else {
- Ok(MaybeTlsStream::Normal(tcp_stream))
- }
- }.boxed()
- } else {
- async move {
- let tcp_stream = connector
- .call(proxy_uri)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", proxy_authority, err))?;
-
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
-
- Ok(MaybeTlsStream::Proxied(tcp_stream))
- }.boxed()
- }
- } else {
- async move {
- let dst_str = dst.to_string(); // for error messages
- let tcp_stream = connector
- .call(dst)
- .await
- .map_err(|err| format_err!("error connecting to {} - {}", dst_str, err))?;
-
- let _ = set_tcp_keepalive(tcp_stream.as_raw_fd(), keepalive);
-
- if is_https {
- Self::secure_stream(tcp_stream, &ssl_connector, &host).await
- } else {
- Ok(MaybeTlsStream::Normal(tcp_stream))
- }
- }.boxed()
- }
- }
-}
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index 729711c8..fa3eadf4 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -7,10 +7,12 @@ use http::{Request, Response, HeaderValue};
use openssl::ssl::{SslConnector, SslMethod};
use futures::*;
-use proxmox_http::http::ProxyConfig;
+use proxmox_http::http::{
+ ProxyConfig,
+ client::HttpsConnector,
+};
use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
-use crate::tools::http::HttpsConnector;
/// Asyncrounous HTTP client implementation
pub struct SimpleHttp {
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (18 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http " Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http Fabian Grünbichler
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
in preparation of moving the abstraction to proxmox_http
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
src/acme/client.rs | 4 +--
src/api2/node/apt.rs | 8 +++--
src/tools.rs | 16 +++++++++
src/tools/simple_http_client.rs | 59 +++++++++++++++++++++------------
src/tools/subscription.rs | 8 +++--
5 files changed, 67 insertions(+), 28 deletions(-)
diff --git a/src/acme/client.rs b/src/acme/client.rs
index 28f277e9..1a6ca46f 100644
--- a/src/acme/client.rs
+++ b/src/acme/client.rs
@@ -19,7 +19,7 @@ use proxmox_acme_rs::{Account, Authorization, Challenge, Directory, Error, Error
use crate::api2::types::AcmeAccountName;
use crate::config::acme::account_path;
-use crate::tools::SimpleHttp;
+use crate::tools::{pbs_simple_http, SimpleHttp};
/// Our on-disk format inherited from PVE's proxmox-acme code.
#[derive(Deserialize, Serialize)]
@@ -72,7 +72,7 @@ impl AcmeClient {
account: None,
directory: None,
nonce: None,
- http_client: SimpleHttp::new(None),
+ http_client: pbs_simple_http(None),
}
}
diff --git a/src/api2/node/apt.rs b/src/api2/node/apt.rs
index 120d5339..9b2d1572 100644
--- a/src/api2/node/apt.rs
+++ b/src/api2/node/apt.rs
@@ -11,7 +11,11 @@ use proxmox_http::http::ProxyConfig;
use crate::config::node;
use crate::server::WorkerTask;
-use crate::tools::{apt, SimpleHttp, subscription};
+use crate::tools::{
+ apt,
+ pbs_simple_http,
+ subscription,
+};
use crate::config::acl::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY};
use crate::api2::types::{Authid, APTUpdateInfo, NODE_SCHEMA, UPID_SCHEMA};
@@ -227,7 +231,7 @@ fn apt_get_changelog(
}
let proxy_config = read_and_update_proxy_config()?;
- let mut client = SimpleHttp::new(proxy_config);
+ let mut client = pbs_simple_http(proxy_config);
let changelog_url = &pkg_info[0].change_log_url;
// FIXME: use 'apt-get changelog' for proxmox packages as well, once repo supports it
diff --git a/src/tools.rs b/src/tools.rs
index eb9a9fd0..4253c054 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -18,6 +18,8 @@ use percent_encoding::{utf8_percent_encode, AsciiSet};
pub use proxmox::tools::fd::Fd;
use proxmox::tools::fs::{create_path, CreateOptions};
+use proxmox_http::http::ProxyConfig;
+
pub mod acl;
pub mod apt;
pub mod async_io;
@@ -34,6 +36,7 @@ pub mod fuse_loop;
mod simple_http_client;
pub use simple_http_client::SimpleHttp;
+pub use simple_http_client::SimpleHttpOptions;
pub mod json;
pub mod logrotate;
@@ -483,6 +486,19 @@ impl<T: Any> AsAny for T {
/// The default 2 hours are far too long for PBS
pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
+pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-backup-client/1.0";
+
+/// Returns a new instance of `SimpleHttp` configured for PBS usage.
+pub fn pbs_simple_http(proxy_config: Option<ProxyConfig>) -> SimpleHttp {
+ let options = SimpleHttpOptions {
+ proxy_config,
+ user_agent: Some(DEFAULT_USER_AGENT_STRING.to_string()),
+ tcp_keepalive: Some(PROXMOX_BACKUP_TCP_KEEPALIVE_TIME),
+ ..Default::default()
+ };
+
+ SimpleHttp::with_options(options)
+}
/// This used to be: `SIMPLE_ENCODE_SET` plus space, `"`, `#`, `<`, `>`, backtick, `?`, `{`, `}`
pub const DEFAULT_ENCODE_SET: &AsciiSet = &percent_encoding::CONTROLS // 0..1f and 7e
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
index fa3eadf4..84e9b69f 100644
--- a/src/tools/simple_http_client.rs
+++ b/src/tools/simple_http_client.rs
@@ -12,50 +12,65 @@ use proxmox_http::http::{
client::HttpsConnector,
};
-use crate::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
+/// Options for a SimpleHttp client.
+#[derive(Default)]
+pub struct SimpleHttpOptions {
+ /// Proxy configuration
+ pub proxy_config: Option<ProxyConfig>,
+ /// `User-Agent` header value, defaults to `proxmox-simple-http-client/0.1`
+ pub user_agent: Option<String>,
+ /// TCP keepalive time, defaults to 7200
+ pub tcp_keepalive: Option<u32>,
+}
+
+impl SimpleHttpOptions {
+ fn get_proxy_authorization(&self) -> Option<String> {
+ if let Some(ref proxy_config) = self.proxy_config {
+ if !proxy_config.force_connect {
+ return proxy_config.authorization.clone();
+ }
+ }
+
+ None
+ }
+}
/// Asyncrounous HTTP client implementation
pub struct SimpleHttp {
client: Client<HttpsConnector, Body>,
- proxy_authorization: Option<String>, // Proxy-Authorization header value
- user_agent: Option<String>,
+ options: SimpleHttpOptions,
}
impl SimpleHttp {
+ pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-simple-http-client/0.1";
- pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-backup-client/1.0";
+ pub fn new() -> Self {
+ Self::with_options(SimpleHttpOptions::default())
+ }
- pub fn new(proxy_config: Option<ProxyConfig>) -> Self {
+ pub fn with_options(options: SimpleHttpOptions) -> Self {
let ssl_connector = SslConnector::builder(SslMethod::tls()).unwrap().build();
- Self::with_ssl_connector(ssl_connector, proxy_config)
+ Self::with_ssl_connector(ssl_connector, options)
}
- pub fn with_ssl_connector(ssl_connector: SslConnector, proxy_config: Option<ProxyConfig>) -> Self {
-
- let mut proxy_authorization = None;
- if let Some(ref proxy_config) = proxy_config {
- if !proxy_config.force_connect {
- proxy_authorization = proxy_config.authorization.clone();
- }
- }
-
+ pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector, PROXMOX_BACKUP_TCP_KEEPALIVE_TIME);
- if let Some(proxy_config) = proxy_config {
- https.set_proxy(proxy_config);
+ let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
+ if let Some(ref proxy_config) = options.proxy_config {
+ https.set_proxy(proxy_config.clone());
}
let client = Client::builder().build(https);
- Self { client, proxy_authorization, user_agent: None }
+ Self { client, options }
}
pub fn set_user_agent(&mut self, user_agent: &str) -> Result<(), Error> {
- self.user_agent = Some(user_agent.to_owned());
+ self.options.user_agent = Some(user_agent.to_owned());
Ok(())
}
fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
- if let Some(ref authorization) = self.proxy_authorization {
+ if let Some(ref authorization) = self.options.get_proxy_authorization() {
request
.headers_mut()
.insert(
@@ -68,7 +83,7 @@ impl SimpleHttp {
}
pub async fn request(&self, mut request: Request<Body>) -> Result<Response<Body>, Error> {
- let user_agent = if let Some(ref user_agent) = self.user_agent {
+ let user_agent = if let Some(ref user_agent) = self.options.user_agent {
HeaderValue::from_str(&user_agent)?
} else {
HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
diff --git a/src/tools/subscription.rs b/src/tools/subscription.rs
index 212cf5e8..ed42e630 100644
--- a/src/tools/subscription.rs
+++ b/src/tools/subscription.rs
@@ -7,7 +7,11 @@ use regex::Regex;
use proxmox::api::api;
use crate::config::node;
-use crate::tools::{self, SimpleHttp};
+use crate::tools::{
+ self,
+ pbs_simple_http,
+ SimpleHttp,
+};
use proxmox::tools::fs::{replace_file, CreateOptions};
/// How long the local key is valid for in between remote checks
@@ -109,7 +113,7 @@ async fn register_subscription(
None
};
- let mut client = SimpleHttp::new(proxy_config);
+ let mut client = pbs_simple_http(proxy_config);
let uri = "https://shop.maurer-it.com/modules/servers/licensing/verify.php";
let query = tools::json_object_to_query(params)?;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
` (19 preceding siblings ...)
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits Fabian Grünbichler
@ 2021-05-14 13:44 ` Fabian Grünbichler
20 siblings, 0 replies; 24+ messages in thread
From: Fabian Grünbichler @ 2021-05-14 13:44 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
Notes:
requires proxmox patch #10
src/acme/client.rs | 3 +-
src/tools.rs | 10 +-
src/tools/simple_http_client.rs | 157 --------------------------------
src/tools/subscription.rs | 2 +-
4 files changed, 8 insertions(+), 164 deletions(-)
delete mode 100644 src/tools/simple_http_client.rs
diff --git a/src/acme/client.rs b/src/acme/client.rs
index 1a6ca46f..d1f46617 100644
--- a/src/acme/client.rs
+++ b/src/acme/client.rs
@@ -16,10 +16,11 @@ use proxmox_acme_rs::account::AccountData as AcmeAccountData;
use proxmox_acme_rs::order::{Order, OrderData};
use proxmox_acme_rs::Request as AcmeRequest;
use proxmox_acme_rs::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
+use proxmox_http::http::client::SimpleHttp;
use crate::api2::types::AcmeAccountName;
use crate::config::acme::account_path;
-use crate::tools::{pbs_simple_http, SimpleHttp};
+use crate::tools::pbs_simple_http;
/// Our on-disk format inherited from PVE's proxmox-acme code.
#[derive(Deserialize, Serialize)]
diff --git a/src/tools.rs b/src/tools.rs
index 4253c054..a9f57b88 100644
--- a/src/tools.rs
+++ b/src/tools.rs
@@ -18,7 +18,11 @@ use percent_encoding::{utf8_percent_encode, AsciiSet};
pub use proxmox::tools::fd::Fd;
use proxmox::tools::fs::{create_path, CreateOptions};
-use proxmox_http::http::ProxyConfig;
+use proxmox_http::http::{
+ client::SimpleHttp,
+ client::SimpleHttpOptions,
+ ProxyConfig,
+};
pub mod acl;
pub mod apt;
@@ -34,10 +38,6 @@ pub mod format;
pub mod fs;
pub mod fuse_loop;
-mod simple_http_client;
-pub use simple_http_client::SimpleHttp;
-pub use simple_http_client::SimpleHttpOptions;
-
pub mod json;
pub mod logrotate;
pub mod loopdev;
diff --git a/src/tools/simple_http_client.rs b/src/tools/simple_http_client.rs
deleted file mode 100644
index 84e9b69f..00000000
--- a/src/tools/simple_http_client.rs
+++ /dev/null
@@ -1,157 +0,0 @@
-use anyhow::{Error, format_err, bail};
-use std::collections::HashMap;
-
-use hyper::Body;
-use hyper::client::{Client, HttpConnector};
-use http::{Request, Response, HeaderValue};
-use openssl::ssl::{SslConnector, SslMethod};
-use futures::*;
-
-use proxmox_http::http::{
- ProxyConfig,
- client::HttpsConnector,
-};
-
-/// Options for a SimpleHttp client.
-#[derive(Default)]
-pub struct SimpleHttpOptions {
- /// Proxy configuration
- pub proxy_config: Option<ProxyConfig>,
- /// `User-Agent` header value, defaults to `proxmox-simple-http-client/0.1`
- pub user_agent: Option<String>,
- /// TCP keepalive time, defaults to 7200
- pub tcp_keepalive: Option<u32>,
-}
-
-impl SimpleHttpOptions {
- fn get_proxy_authorization(&self) -> Option<String> {
- if let Some(ref proxy_config) = self.proxy_config {
- if !proxy_config.force_connect {
- return proxy_config.authorization.clone();
- }
- }
-
- None
- }
-}
-
-/// Asyncrounous HTTP client implementation
-pub struct SimpleHttp {
- client: Client<HttpsConnector, Body>,
- options: SimpleHttpOptions,
-}
-
-impl SimpleHttp {
- pub const DEFAULT_USER_AGENT_STRING: &'static str = "proxmox-simple-http-client/0.1";
-
- pub fn new() -> Self {
- Self::with_options(SimpleHttpOptions::default())
- }
-
- pub fn with_options(options: SimpleHttpOptions) -> Self {
- let ssl_connector = SslConnector::builder(SslMethod::tls()).unwrap().build();
- Self::with_ssl_connector(ssl_connector, options)
- }
-
- pub fn with_ssl_connector(ssl_connector: SslConnector, options: SimpleHttpOptions) -> Self {
- let connector = HttpConnector::new();
- let mut https = HttpsConnector::with_connector(connector, ssl_connector, options.tcp_keepalive.unwrap_or(7200));
- if let Some(ref proxy_config) = options.proxy_config {
- https.set_proxy(proxy_config.clone());
- }
- let client = Client::builder().build(https);
- Self { client, options }
- }
-
- pub fn set_user_agent(&mut self, user_agent: &str) -> Result<(), Error> {
- self.options.user_agent = Some(user_agent.to_owned());
- Ok(())
- }
-
- fn add_proxy_headers(&self, request: &mut Request<Body>) -> Result<(), Error> {
- if request.uri().scheme() != Some(&http::uri::Scheme::HTTPS) {
- if let Some(ref authorization) = self.options.get_proxy_authorization() {
- request
- .headers_mut()
- .insert(
- http::header::PROXY_AUTHORIZATION,
- HeaderValue::from_str(authorization)?,
- );
- }
- }
- Ok(())
- }
-
- pub async fn request(&self, mut request: Request<Body>) -> Result<Response<Body>, Error> {
- let user_agent = if let Some(ref user_agent) = self.options.user_agent {
- HeaderValue::from_str(&user_agent)?
- } else {
- HeaderValue::from_str(Self::DEFAULT_USER_AGENT_STRING)?
- };
-
- request.headers_mut().insert(hyper::header::USER_AGENT, user_agent);
-
- self.add_proxy_headers(&mut request)?;
-
- self.client.request(request)
- .map_err(Error::from)
- .await
- }
-
- pub async fn post(
- &mut self,
- uri: &str,
- body: Option<String>,
- content_type: Option<&str>,
- ) -> Result<Response<Body>, Error> {
-
- let body = if let Some(body) = body {
- Body::from(body)
- } else {
- Body::empty()
- };
- let content_type = content_type.unwrap_or("application/json");
-
- let request = Request::builder()
- .method("POST")
- .uri(uri)
- .header(hyper::header::CONTENT_TYPE, content_type)
- .body(body)?;
-
- self.request(request).await
- }
-
- pub async fn get_string(
- &mut self,
- uri: &str,
- extra_headers: Option<&HashMap<String, String>>,
- ) -> Result<String, Error> {
-
- let mut request = Request::builder()
- .method("GET")
- .uri(uri);
-
- if let Some(hs) = extra_headers {
- for (h, v) in hs.iter() {
- request = request.header(h, v);
- }
- }
-
- let request = request.body(Body::empty())?;
-
- let res = self.request(request).await?;
-
- let status = res.status();
- if !status.is_success() {
- bail!("Got bad status '{}' from server", status)
- }
-
- Self::response_body_string(res).await
- }
-
- pub async fn response_body_string(res: Response<Body>) -> Result<String, Error> {
- let buf = hyper::body::to_bytes(res).await?;
- String::from_utf8(buf.to_vec())
- .map_err(|err| format_err!("Error converting HTTP result data: {}", err))
- }
-}
diff --git a/src/tools/subscription.rs b/src/tools/subscription.rs
index ed42e630..c549c3dd 100644
--- a/src/tools/subscription.rs
+++ b/src/tools/subscription.rs
@@ -10,9 +10,9 @@ use crate::config::node;
use crate::tools::{
self,
pbs_simple_http,
- SimpleHttp,
};
use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_http::http::client::SimpleHttp;
/// How long the local key is valid for in between remote checks
pub const MAX_LOCAL_KEY_AGE: i64 = 15 * 24 * 3600;
--
2.20.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] applied: [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
@ 2021-05-17 8:22 ` Dietmar Maurer
0 siblings, 0 replies; 24+ messages in thread
From: Dietmar Maurer @ 2021-05-17 8:22 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
applied all 13 patches
On 5/14/21 3:44 PM, Fabian Grünbichler wrote:
> this is shadowed by the websocket feature being enabled by default.
>
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
>
> Notes:
> fixes issue uncovered by next patch
>
> proxmox/Cargo.toml | 2 +-
> proxmox/debian/control | 1 +
> 2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/proxmox/Cargo.toml b/proxmox/Cargo.toml
> index c04c105..da46e74 100644
> --- a/proxmox/Cargo.toml
> +++ b/proxmox/Cargo.toml
> @@ -57,7 +57,7 @@ sortable-macro = ["proxmox-sortable-macro"]
> api-macro = ["proxmox-api-macro"]
> test-harness = []
> cli = [ "router", "hyper", "tokio" ]
> -router = [ "hyper", "tokio" ]
> +router = [ "futures", "hyper", "tokio" ]
> websocket = [ "futures", "hyper", "openssl", "tokio/sync", "tokio/io-util", "openssl" ]
> tfa = [ "openssl" ]
> u2f = [ "base32" ]
> diff --git a/proxmox/debian/control b/proxmox/debian/control
> index da55b88..9091cb8 100644
> --- a/proxmox/debian/control
> +++ b/proxmox/debian/control
> @@ -267,6 +267,7 @@ Multi-Arch: same
> Depends:
> ${misc:Depends},
> librust-proxmox-dev (= ${binary:Version}),
> + librust-futures-0.3+default-dev,
> librust-hyper-0.14+default-dev,
> librust-hyper-0.14+full-dev,
> librust-tokio-1+default-dev
^ permalink raw reply [flat|nested] 24+ messages in thread
* [pbs-devel] applied: [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
@ 2021-05-17 8:35 ` Dietmar Maurer
0 siblings, 0 replies; 24+ messages in thread
From: Dietmar Maurer @ 2021-05-17 8:35 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
applied all 8 patches
On 5/14/21 3:44 PM, Fabian Grünbichler wrote:
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
>
> Notes:
> requires patch #3 in proxmox
>
> Cargo.toml | 5 +++--
> debian/control | 3 ++-
> src/api2/node.rs | 2 +-
> 3 files changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/Cargo.toml b/Cargo.toml
> index cd99241e..997017c0 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -53,10 +53,11 @@ percent-encoding = "2.1"
> pin-utils = "0.1.0"
> pin-project = "1.0"
> pathpatterns = "0.1.2"
> -proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro", "websocket" ] }
> +proxmox = { version = "0.11.4", features = [ "sortable-macro", "api-macro" ] }
> #proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
> -#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro", "websocket" ] }
> +#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro" ] }
> proxmox-fuse = "0.1.1"
> +proxmox-http = { version = "0.1.0", path = "../proxmox/proxmox-http", features = [ "websocket" ] }
> pxar = { version = "0.10.1", features = [ "tokio-io" ] }
> #pxar = { path = "../pxar", features = [ "tokio-io" ] }
> regex = "1.2"
> diff --git a/debian/control b/debian/control
> index 3f1edba8..a02518fe 100644
> --- a/debian/control
> +++ b/debian/control
> @@ -42,9 +42,10 @@ Build-Depends: debhelper (>= 11),
> librust-proxmox-0.11+api-macro-dev (>= 0.11.4-~~),
> librust-proxmox-0.11+default-dev (>= 0.11.4-~~),
> librust-proxmox-0.11+sortable-macro-dev (>= 0.11.4-~~),
> - librust-proxmox-0.11+websocket-dev (>= 0.11.4-~~),
> librust-proxmox-acme-rs-0.2+default-dev (>= 0.2.1-~~),
> librust-proxmox-fuse-0.1+default-dev (>= 0.1.1-~~),
> + librust-proxmox-http-0.1+default-dev,
> + librust-proxmox-http-0.1+websocket-dev,
> librust-pxar-0.10+default-dev (>= 0.10.1-~~),
> librust-pxar-0.10+tokio-io-dev (>= 0.10.1-~~),
> librust-regex-1+default-dev (>= 1.2-~~),
> diff --git a/src/api2/node.rs b/src/api2/node.rs
> index 75271cd5..af9ecf69 100644
> --- a/src/api2/node.rs
> +++ b/src/api2/node.rs
> @@ -17,7 +17,7 @@ use proxmox::api::{
> api, schema::*, ApiHandler, ApiMethod, ApiResponseFuture, Permission, RpcEnvironment,
> };
> use proxmox::list_subdirs_api_method;
> -use proxmox::tools::websocket::WebSocket;
> +use proxmox_http::websocket::WebSocket;
> use proxmox::{identity, sortable};
>
> use crate::api2::types::*;
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2021-05-17 8:35 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-14 13:44 [pbs-devel] [PATCH proxmox(-backup) 00/21] pull HTTP code into proxmox/proxmox-http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 01/13] proxmox: add missing +router -> futures dep Fabian Grünbichler
2021-05-17 8:22 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 02/13] meta: add empty proxmox-http sub-crate Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 03/13] http: takeover websocket feature from proxmox Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 04/13] http: make clippy happy Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 05/13] proxmox: takeover socket helper from proxmox_backup Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 06/13] http: takeover MaybeTlsStream " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 07/13] http: takeover build_authority helper " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 08/13] http: takeover ProxyConfig " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 09/13] http: takeover tools::http " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 10/13] http: takeover simple HTTP client " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 11/13] http: make clippy happy Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 12/13] http: rustfmt Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox 13/13] http: update d/control Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 1/8] move websocket to new 'proxmox_http' crate Fabian Grünbichler
2021-05-17 8:35 ` [pbs-devel] applied: " Dietmar Maurer
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 2/8] refactor: move socket helper to proxmox crate Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 3/8] move MaybeTlsStream wrapper to proxmox_http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 4/8] HttpsConnector: make keepalive configurable Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 5/8] move ProxyConfig to proxmox_http Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 6/8] move tools::http " Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 7/8] SimpleHttp: factor out product-specific bits Fabian Grünbichler
2021-05-14 13:44 ` [pbs-devel] [PATCH proxmox-backup 8/8] move SimpleHttp to proxmox_http Fabian Grünbichler
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal