[pbs-devel] [PATCH proxmox-backup v2] daemon: add hack for sd_notify

[pbs-devel] [PATCH proxmox-backup v2] daemon: add hack for sd_notify

[Dominik Csapak]

sd_notify is not synchronous, iow. it only waits until the message
reaches the queue not until it is processed by systemd

when the process that sent such a message exits before systemd could
process it, it cannot be associated to the correct pid

so in case of reloading, we send a message with 'MAINPID=<newpid>'
to signal that it will change. if now the old process exits before
systemd knows this, it will not accept the 'READY=1' message from the
child, since it rejects the MAINPID change

since there is no (AFAICS) library interface to check the unit status,
we use 'systemctl is-active <SERVICE_NAME>' to check the state until
it is not 'reloading' anymore.

on newer systemd versions, there is 'sd_notify_barrier' which would
allow us to wait for systemd to have all messages from the current
pid to be processed before acknowledging to the child, but on buster
the systemd version is to old...

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
changes from v1:
* only check the state in the parent pid, do not send ready
  in the child in a loop
* changed the comment to include "FIXME"
 src/bin/proxmox-backup-api.rs   |  1 +
 src/bin/proxmox-backup-proxy.rs |  1 +
 src/tools/daemon.rs             | 26 ++++++++++++++++++++++++++
 3 files changed, 28 insertions(+)

diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index 7d59717b..70d4cb5d 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -76,6 +76,7 @@ async fn run() -> Result<(), Error> {
                 })
             )
         },
+        "proxmox-backup.service",
     );
 
     server::write_pid(buildcfg::PROXMOX_BACKUP_API_PID_FN)?;
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 04c976b5..259d558a 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -133,6 +133,7 @@ async fn run() -> Result<(), Error> {
                 .map(|_| ())
             )
         },
+        "proxmox-backup-proxy.service",
     );
 
     server::write_pid(buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)?;
diff --git a/src/tools/daemon.rs b/src/tools/daemon.rs
index 249ce2ad..63eb6dee 100644
--- a/src/tools/daemon.rs
+++ b/src/tools/daemon.rs
@@ -260,6 +260,7 @@ impl Future for NotifyReady {
 pub async fn create_daemon<F, S>(
     address: std::net::SocketAddr,
     create_service: F,
+    service_name: &str,
 ) -> Result<(), Error>
 where
     F: FnOnce(tokio::net::TcpListener, NotifyReady) -> Result<S, Error>,
@@ -301,10 +302,35 @@ where
     if let Some(future) = finish_future {
         future.await;
     }
+
+    // FIXME: this is a hack, replace with sd_notify_barrier when available
+    if server::is_reload_request() {
+        check_service_is_active(service_name).await?;
+    }
+
     log::info!("daemon shut down...");
     Ok(())
 }
 
+pub async fn check_service_is_active(service: &str) -> Result<(), Error> {
+    for _ in 0..5 {
+        tokio::time::delay_for(std::time::Duration::new(5, 0)).await;
+        if let Ok(output) = tokio::process::Command::new("systemctl")
+            .args(&["is-active", service])
+            .output()
+            .await
+        {
+            if let Ok(text) = String::from_utf8(output.stdout) {
+                if text.trim().trim_start() != "reloading" {
+                    return Ok(());
+                }
+            }
+        }
+    }
+
+    Ok(())
+}
+
 #[link(name = "systemd")]
 extern "C" {
     fn sd_notify(unset_environment: c_int, state: *const c_char) -> c_int;
-- 
2.20.1

[pbs-devel] applied: [PATCH proxmox-backup v2] daemon: add hack for sd_notify

[Thomas Lamprecht]

On 11.11.20 09:27, Dominik Csapak wrote:
> sd_notify is not synchronous, iow. it only waits until the message
> reaches the queue not until it is processed by systemd
> 
> when the process that sent such a message exits before systemd could
> process it, it cannot be associated to the correct pid
> 
> so in case of reloading, we send a message with 'MAINPID=<newpid>'
> to signal that it will change. if now the old process exits before
> systemd knows this, it will not accept the 'READY=1' message from the
> child, since it rejects the MAINPID change
> 
> since there is no (AFAICS) library interface to check the unit status,
> we use 'systemctl is-active <SERVICE_NAME>' to check the state until
> it is not 'reloading' anymore.
> 
> on newer systemd versions, there is 'sd_notify_barrier' which would
> allow us to wait for systemd to have all messages from the current
> pid to be processed before acknowledging to the child, but on buster
> the systemd version is to old...
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> changes from v1:
> * only check the state in the parent pid, do not send ready
>   in the child in a loop
> * changed the comment to include "FIXME"
>  src/bin/proxmox-backup-api.rs   |  1 +
>  src/bin/proxmox-backup-proxy.rs |  1 +
>  src/tools/daemon.rs             | 26 ++++++++++++++++++++++++++
>  3 files changed, 28 insertions(+)
> 
>

applied, thanks! With the followups we spoke off-list.