[pbs-devel] [PATCH proxmox-backup v3 0/3] add rest_server example

[pbs-devel] [PATCH proxmox-backup v3 0/3] add rest_server example

[Dominik Csapak]

adds a rest_server example to the proxmox-rest-server crate.

changes from v2:
* don't use pbs_runtime, but use a stock tokio runtime (cuts the
  pbs-runtime dependency)
* improve commit messages (explain hyper addrincoming, and api call
  paths)
* use proxmox "0.13" dev-dependency instead of "0.13.5"
* use explicit hyper version dependency (0.14.5)

changes from v1:
* added patches so we can remove the (now) unused srt/tools/async_io.rs.
  this enables us to don't have any dependencies on the top-level crate
  for the example, and we can move it into proxmox-rest-server
* use hypers AddrStream for the example

Dominik Csapak (4):
  rest-server: use hypers AddrIncoming for proxmox-backup-api
  remove tools/async_io.rs
  examples: add example for a simple rest server with a small api
  fixup! rest-server: use hypers AddrIncoming for proxmox-backup-api

 proxmox-rest-server/Cargo.toml              |   7 +-
 proxmox-rest-server/examples/rest_server.rs | 220 ++++++++++++++++++++
 proxmox-rest-server/src/rest.rs             |  18 +-
 src/bin/proxmox-backup-api.rs               |   3 +-
 src/tools/async_io.rs                       |  83 --------
 src/tools/mod.rs                            |   1 -
 6 files changed, 236 insertions(+), 96 deletions(-)
 create mode 100644 proxmox-rest-server/examples/rest_server.rs
 delete mode 100644 src/tools/async_io.rs

-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup v3 1/4] rest-server: use hypers AddrIncoming for proxmox-backup-api

[Dominik Csapak]

this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).

this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.

The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox-rest-server/src/rest.rs | 18 ++++++++----------
 src/bin/proxmox-backup-api.rs   |  3 ++-
 2 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/proxmox-rest-server/src/rest.rs b/proxmox-rest-server/src/rest.rs
index ba2edf11..8c1e34a3 100644
--- a/proxmox-rest-server/src/rest.rs
+++ b/proxmox-rest-server/src/rest.rs
@@ -93,7 +93,7 @@ impl tower_service::Service<&Pin<Box<tokio_openssl::SslStream<tokio::net::TcpStr
     }
 }
 
-impl tower_service::Service<&tokio::net::TcpStream> for RestServer {
+impl tower_service::Service<&hyper::server::conn::AddrStream> for RestServer {
     type Response = ApiService;
     type Error = Error;
     type Future = Pin<Box<dyn Future<Output = Result<ApiService, Error>> + Send>>;
@@ -102,15 +102,13 @@ impl tower_service::Service<&tokio::net::TcpStream> for RestServer {
         Poll::Ready(Ok(()))
     }
 
-    fn call(&mut self, ctx: &tokio::net::TcpStream) -> Self::Future {
-        match ctx.peer_addr() {
-            Err(err) => future::err(format_err!("unable to get peer address - {}", err)).boxed(),
-            Ok(peer) => future::ok(ApiService {
-                peer,
-                api_config: self.api_config.clone(),
-            })
-            .boxed(),
-        }
+    fn call(&mut self, ctx: &hyper::server::conn::AddrStream) -> Self::Future {
+        let peer =  ctx.remote_addr();
+        future::ok(ApiService {
+            peer,
+            api_config: self.api_config.clone(),
+        })
+        .boxed()
     }
 }
 
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index 86650de6..e3f2531f 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -108,7 +108,8 @@ async fn run() -> Result<(), Error> {
     let server = daemon::create_daemon(
         ([127,0,0,1], 82).into(),
         move |listener, ready| {
-            let incoming = proxmox_backup::tools::async_io::StaticIncoming::from(listener);
+            let incoming = hyper::server::conn::AddrIncoming::from_listener(listener)?;
+
             Ok(ready
                 .and_then(|_| hyper::Server::builder(incoming)
                     .serve(rest_server)
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup v3 2/4] remove tools/async_io.rs

[Dominik Csapak]

nothing from here is used anymore, so remove it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/tools/async_io.rs | 83 -------------------------------------------
 src/tools/mod.rs      |  1 -
 2 files changed, 84 deletions(-)
 delete mode 100644 src/tools/async_io.rs

diff --git a/src/tools/async_io.rs b/src/tools/async_io.rs
deleted file mode 100644
index 66d38094..00000000
--- a/src/tools/async_io.rs
+++ /dev/null
@@ -1,83 +0,0 @@
-//! AsyncRead/AsyncWrite utilities.
-
-use std::os::unix::io::{AsRawFd, RawFd};
-use std::pin::Pin;
-use std::task::{Context, Poll};
-
-use futures::stream::{Stream, TryStream};
-use futures::ready;
-use tokio::io::{AsyncRead, AsyncWrite};
-use tokio::net::TcpListener;
-
-
-/// Tokio's `Incoming` now is a reference type and hyper's `AddrIncoming` misses some standard
-/// stuff like `AsRawFd`, so here's something implementing hyper's `Accept` from a `TcpListener`
-pub struct StaticIncoming(TcpListener);
-
-impl From<TcpListener> for StaticIncoming {
-    fn from(inner: TcpListener) -> Self {
-        Self(inner)
-    }
-}
-
-impl AsRawFd for StaticIncoming {
-    fn as_raw_fd(&self) -> RawFd {
-        self.0.as_raw_fd()
-    }
-}
-
-impl hyper::server::accept::Accept for StaticIncoming {
-    type Conn = tokio::net::TcpStream;
-    type Error = std::io::Error;
-
-    fn poll_accept(
-        self: Pin<&mut Self>,
-        cx: &mut Context,
-    ) -> Poll<Option<Result<Self::Conn, Self::Error>>> {
-        let this = self.get_mut();
-        loop {
-            match ready!(this.0.poll_accept(cx)) {
-                Ok((conn, _addr)) => return Poll::Ready(Some(Ok(conn))),
-                Err(err) => {
-                    eprintln!("error accepting connection: {}", err);
-                    continue;
-                }
-            }
-        }
-    }
-}
-
-/// We also implement TryStream for this, as tokio doesn't do this anymore either and we want to be
-/// able to map connections to then add eg. ssl to them. This support code makes the changes
-/// required for hyper 0.13 a bit less annoying to read.
-impl Stream for StaticIncoming {
-    type Item = std::io::Result<(tokio::net::TcpStream, std::net::SocketAddr)>;
-
-    fn poll_next(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Option<Self::Item>> {
-        match self.get_mut().0.poll_accept(cx) {
-            Poll::Pending => Poll::Pending,
-            Poll::Ready(result) => Poll::Ready(Some(result)),
-        }
-    }
-}
-
-/// Implement hyper's `Accept` for any `TryStream` of sockets:
-pub struct HyperAccept<T>(pub T);
-
-
-impl<T, I> hyper::server::accept::Accept for HyperAccept<T>
-where
-    T: TryStream<Ok = I> + Unpin,
-    I: AsyncRead + AsyncWrite,
-{
-    type Conn = I;
-    type Error = T::Error;
-
-    fn poll_accept(
-        self: Pin<&mut Self>,
-        cx: &mut Context,
-    ) -> Poll<Option<Result<Self::Conn, Self::Error>>> {
-        let this = Pin::new(&mut self.get_mut().0);
-        this.try_poll_next(cx)
-    }
-}
diff --git a/src/tools/mod.rs b/src/tools/mod.rs
index 5dc129f0..d8ed7275 100644
--- a/src/tools/mod.rs
+++ b/src/tools/mod.rs
@@ -13,7 +13,6 @@ use proxmox_http::{
 };
 
 pub mod apt;
-pub mod async_io;
 pub mod config;
 pub mod disks;
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup v3 3/4] examples: add example for a simple rest server with a small api

[Dominik Csapak]

show how to generally start a daemon that serves a rest api + index page

api calls are (prefixed with either /api2/json or /api2/extjs):
/		GET	listing
/ping		GET	returns "pong"
/items		GET	lists existing items
		POST	lets user create new items
/items/{id}	GET	returns the content of a single item
		PUT	updates an item
		DELETE	deletes an item

Contains a small dummy user/authinfo

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox-rest-server/Cargo.toml              |   5 +
 proxmox-rest-server/examples/rest_server.rs | 220 ++++++++++++++++++++
 2 files changed, 225 insertions(+)
 create mode 100644 proxmox-rest-server/examples/rest_server.rs

diff --git a/proxmox-rest-server/Cargo.toml b/proxmox-rest-server/Cargo.toml
index b0e53d19..dedeef73 100644
--- a/proxmox-rest-server/Cargo.toml
+++ b/proxmox-rest-server/Cargo.toml
@@ -5,6 +5,11 @@ authors = ["Proxmox Support Team <support@proxmox.com>"]
 edition = "2018"
 description = "REST server implementation"
 
+# for example
+[dev-dependencies]
+proxmox = { version = "0.13", features = ["router","api-macro"] }
+tokio = { version = "1.6", features = [ "rt-multi-thread", "signal", "process" ] }
+
 [dependencies]
 anyhow = "1.0"
 futures = "0.3"
diff --git a/proxmox-rest-server/examples/rest_server.rs b/proxmox-rest-server/examples/rest_server.rs
new file mode 100644
index 00000000..4d0dac35
--- /dev/null
+++ b/proxmox-rest-server/examples/rest_server.rs
@@ -0,0 +1,220 @@
+use std::sync::{Arc, Mutex};
+use std::collections::HashMap;
+
+use anyhow::{bail, format_err, Error};
+use futures::{FutureExt, TryFutureExt};
+use lazy_static::lazy_static;
+
+use proxmox::api::{api, router::SubdirMap, Router, RpcEnvironmentType, UserInformation};
+use proxmox::list_subdirs_api_method;
+use proxmox_rest_server::{ApiAuth, ApiConfig, AuthError, RestServer};
+
+// Create a Dummy User info and auth system
+// Normally this would check and authenticate the user
+struct DummyUserInfo;
+
+impl UserInformation for DummyUserInfo {
+    fn is_superuser(&self, _userid: &str) -> bool {
+        true
+    }
+    fn is_group_member(&self, _userid: &str, group: &str) -> bool {
+        group == "Group"
+    }
+    fn lookup_privs(&self, _userid: &str, _path: &[&str]) -> u64 {
+        u64::MAX
+    }
+}
+
+struct DummyAuth;
+
+impl ApiAuth for DummyAuth {
+    fn check_auth(
+        &self,
+        _headers: &http::HeaderMap,
+        _method: &hyper::Method,
+    ) -> Result<(String, Box<dyn UserInformation + Sync + Send>), AuthError> {
+        // get some global/cached userinfo
+        let userinfo = DummyUserInfo;
+        // Do some user checks, e.g. cookie/csrf
+        Ok(("User".to_string(), Box::new(userinfo)))
+    }
+}
+
+// this should return the index page of the webserver
+// iow. what the user browses to
+
+fn get_index(
+    _auth_id: Option<String>,
+    _language: Option<String>,
+    _api: &ApiConfig,
+    _parts: http::request::Parts,
+) -> http::Response<hyper::Body> {
+    // build an index page
+    http::Response::builder()
+        .body("hello world".into())
+        .unwrap()
+}
+
+// a few examples on how to do api calls with the Router
+
+#[api]
+/// A simple ping method. returns "pong"
+fn ping() -> Result<String, Error> {
+    Ok("pong".to_string())
+}
+
+lazy_static! {
+    static ref ITEM_MAP: Mutex<HashMap<String, String>> = Mutex::new(HashMap::new());
+}
+
+#[api]
+/// Lists all current items
+fn list_items() -> Result<Vec<String>, Error> {
+    Ok(ITEM_MAP.lock().unwrap().keys().map(|k| k.clone()).collect())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                type: String,
+                description: "The name",
+            },
+            value: {
+                type: String,
+                description: "The value",
+            },
+        },
+    },
+)]
+/// creates a new item
+fn create_item(name: String, value: String) -> Result<(), Error> {
+    let mut map = ITEM_MAP.lock().unwrap();
+    if map.contains_key(&name) {
+        bail!("{} already exists", name);
+    }
+
+    map.insert(name, value);
+
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                type: String,
+                description: "The name",
+            },
+        },
+    },
+)]
+/// returns the value of an item
+fn get_item(name: String) -> Result<String, Error> {
+    ITEM_MAP.lock().unwrap().get(&name).map(|s| s.to_string()).ok_or_else(|| format_err!("no such item '{}'", name))
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                type: String,
+                description: "The name",
+            },
+            value: {
+                type: String,
+                description: "The value",
+            },
+        },
+    },
+)]
+/// updates an item
+fn update_item(name: String, value: String) -> Result<(), Error> {
+    if let Some(val) = ITEM_MAP.lock().unwrap().get_mut(&name) {
+        *val = value;
+    } else {
+        bail!("no such item '{}'", name);
+    }
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            name: {
+                type: String,
+                description: "The name",
+            },
+        },
+    },
+)]
+/// deletes an item
+fn delete_item(name: String) -> Result<(), Error> {
+    if ITEM_MAP.lock().unwrap().remove(&name).is_none() {
+        bail!("no such item '{}'", name);
+    }
+    Ok(())
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_GET_ITEM)
+    .put(&API_METHOD_UPDATE_ITEM)
+    .delete(&API_METHOD_DELETE_ITEM);
+
+const SUBDIRS: SubdirMap = &[
+    (
+        "items",
+        &Router::new()
+            .get(&API_METHOD_LIST_ITEMS)
+            .post(&API_METHOD_CREATE_ITEM)
+            .match_all("name", &ITEM_ROUTER)
+    ),
+    (
+        "ping",
+        &Router::new()
+            .get(&API_METHOD_PING)
+    ),
+];
+
+const ROUTER: Router = Router::new()
+    .get(&list_subdirs_api_method!(SUBDIRS))
+    .subdirs(SUBDIRS);
+
+async fn run() -> Result<(), Error> {
+
+    // we first have to configure the api environment (basedir etc.)
+
+    let config = ApiConfig::new(
+        "/var/tmp/",
+        &ROUTER,
+        RpcEnvironmentType::PUBLIC,
+        Arc::new(DummyAuth {}),
+        get_index,
+    )?;
+    let rest_server = RestServer::new(config);
+
+    // then we have to create a daemon that listens, accepts and serves
+    // the api to clients
+    proxmox_rest_server::daemon::create_daemon(
+        ([127, 0, 0, 1], 65000).into(),
+        move |listener, ready| {
+            let incoming = hyper::server::conn::AddrIncoming::from_listener(listener)?;
+
+            Ok(ready
+                .and_then(|_| hyper::Server::builder(incoming)
+                    .serve(rest_server)
+                    .map_err(Error::from)
+                )
+                .map_err(|err| eprintln!("ERR: {}", err))
+                .map(|test| println!("OK: {}", test.is_ok())))
+        },
+        "example_server",
+    ).await?;
+
+    Ok(())
+}
+
+fn main() -> Result<(), Error> {
+    let rt = tokio::runtime::Runtime::new()?;
+    rt.block_on(async { run().await })
+}
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup v3 4/4] fixup! rest-server: use hypers AddrIncoming for proxmox-backup-api

[Dominik Csapak]

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox-rest-server/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proxmox-rest-server/Cargo.toml b/proxmox-rest-server/Cargo.toml
index dedeef73..8fa1a79f 100644
--- a/proxmox-rest-server/Cargo.toml
+++ b/proxmox-rest-server/Cargo.toml
@@ -15,7 +15,7 @@ anyhow = "1.0"
 futures = "0.3"
 handlebars = "3.0"
 http = "0.2"
-hyper = { version = "0.14", features = [ "full" ] }
+hyper = { version = "0.14.5", features = [ "full" ] }
 lazy_static = "1.4"
 libc = "0.2"
 log = "0.4"
-- 
2.30.2

Re: [pbs-devel] [PATCH proxmox-backup v3 4/4] fixup! rest-server: use hypers AddrIncoming for proxmox-backup-api

[Dominik Csapak]

sorry i did miss a
'git rebase -i master --autosquash' before sending... should i send a v4 
for that?

[pbs-devel] applied-series: [PATCH proxmox-backup v3 0/3] add rest_server example

[Thomas Lamprecht]

On 29.09.21 09:04, Dominik Csapak wrote:
> adds a rest_server example to the proxmox-rest-server crate.
> 
> changes from v2:
> * don't use pbs_runtime, but use a stock tokio runtime (cuts the
>   pbs-runtime dependency)
> * improve commit messages (explain hyper addrincoming, and api call
>   paths)
> * use proxmox "0.13" dev-dependency instead of "0.13.5"
> * use explicit hyper version dependency (0.14.5)
> 
> changes from v1:
> * added patches so we can remove the (now) unused srt/tools/async_io.rs.
>   this enables us to don't have any dependencies on the top-level crate
>   for the example, and we can move it into proxmox-rest-server
> * use hypers AddrStream for the example
> 
> Dominik Csapak (4):
>   rest-server: use hypers AddrIncoming for proxmox-backup-api
>   remove tools/async_io.rs
>   examples: add example for a simple rest server with a small api
>   fixup! rest-server: use hypers AddrIncoming for proxmox-backup-api
> 
>  proxmox-rest-server/Cargo.toml              |   7 +-
>  proxmox-rest-server/examples/rest_server.rs | 220 ++++++++++++++++++++
>  proxmox-rest-server/src/rest.rs             |  18 +-
>  src/bin/proxmox-backup-api.rs               |   3 +-
>  src/tools/async_io.rs                       |  83 --------
>  src/tools/mod.rs                            |   1 -
>  6 files changed, 236 insertions(+), 96 deletions(-)
>  create mode 100644 proxmox-rest-server/examples/rest_server.rs
>  delete mode 100644 src/tools/async_io.rs
> 



applied, thanks! FYI: I `rebase --fixup`'d the series and renamed the example to
minimal-rest-server for now.