[pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Gabriel Goller]

When using the `proxmox-backup-client mount` command, the parent sometimes
exits before we can print any error message. Most notably this happens
when no PBS_REPOSITORY is passed, as this is the first option checked.
Flush the stdout and stderr so that we can guarantee the output.

Reported-by: Friedrich Weber <f.weber@proxmox.com>
Suggested-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
---

v2, thanks @Christian:
 - Removed unneeded pw.try_clone()

 proxmox-backup-client/src/mount.rs | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/proxmox-backup-client/src/mount.rs b/proxmox-backup-client/src/mount.rs
index b69e7e80..e7718b84 100644
--- a/proxmox-backup-client/src/mount.rs
+++ b/proxmox-backup-client/src/mount.rs
@@ -1,6 +1,7 @@
 use std::collections::HashMap;
 use std::ffi::OsStr;
 use std::hash::BuildHasher;
+use std::io::{self, Write};
 use std::os::unix::io::{AsRawFd, OwnedFd};
 use std::path::{Path, PathBuf};
 use std::sync::Arc;
@@ -189,7 +190,10 @@ fn mount(
         Ok(ForkResult::Child) => {
             drop(pr);
             nix::unistd::setsid().unwrap();
-            proxmox_async::runtime::main(mount_do(param, Some(pw)))
+            let result = proxmox_async::runtime::main(mount_do(param, Some(pw)));
+            io::stdout().flush()?;
+            io::stderr().flush()?;
+            result
         }
         Err(_) => bail!("failed to daemonize process"),
     }
-- 
2.43.0



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Christian Ebner]

On 6/14/24 11:39, Gabriel Goller wrote:
> When using the `proxmox-backup-client mount` command, the parent sometimes
> exits before we can print any error message. Most notably this happens
> when no PBS_REPOSITORY is passed, as this is the first option checked.
> Flush the stdout and stderr so that we can guarantee the output.
> 
> Reported-by: Friedrich Weber <f.weber@proxmox.com>
> Suggested-by: Christian Ebner <c.ebner@proxmox.com>
> Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
> ---
> 
> v2, thanks @Christian:
>   - Removed unneeded pw.try_clone()
> 
>   proxmox-backup-client/src/mount.rs | 6 +++++-
>   1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/proxmox-backup-client/src/mount.rs b/proxmox-backup-client/src/mount.rs
> index b69e7e80..e7718b84 100644
> --- a/proxmox-backup-client/src/mount.rs
> +++ b/proxmox-backup-client/src/mount.rs
> @@ -1,6 +1,7 @@
>   use std::collections::HashMap;
>   use std::ffi::OsStr;
>   use std::hash::BuildHasher;
> +use std::io::{self, Write};
>   use std::os::unix::io::{AsRawFd, OwnedFd};
>   use std::path::{Path, PathBuf};
>   use std::sync::Arc;
> @@ -189,7 +190,10 @@ fn mount(
>           Ok(ForkResult::Child) => {
>               drop(pr);
>               nix::unistd::setsid().unwrap();
> -            proxmox_async::runtime::main(mount_do(param, Some(pw)))
> +            let result = proxmox_async::runtime::main(mount_do(param, Some(pw)));
> +            io::stdout().flush()?;
> +            io::stderr().flush()?;
> +            result
>           }
>           Err(_) => bail!("failed to daemonize process"),
>       }

Looks good to me now, consider:

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Friedrich Weber]

Thanks for tackling this! I ran into this issue (sometimes missing error
output) when I was running `sudo proxmox-backup-client mount [...]` and
forgot that PBS_REPOSITORY was not passed through by sudo. It looks like
in this case, the patch doesn't completely fix this issue -- seems like
I'm still missing the error output sometimes:

$ sudo ./proxmox-backup-client mount \
        "$SNAPSHOT" $FILE $PATH \
        --keyfile $KEYFILE --ns $NS
$ sudo ./proxmox-backup-client mount \
        "$SNAPSHOT" $FILE $PATH \
        --keyfile $KEYFILE --ns $NS
$ sudo ./proxmox-backup-client mount \
        "$SNAPSHOT" $FILE $PATH \
        --keyfile $KEYFILE --ns $NS
Error: unable to get (default) repository

On 14/06/2024 11:39, Gabriel Goller wrote:
> When using the `proxmox-backup-client mount` command, the parent sometimes
> exits before we can print any error message. Most notably this happens
> when no PBS_REPOSITORY is passed, as this is the first option checked.
> Flush the stdout and stderr so that we can guarantee the output.
> 
> Reported-by: Friedrich Weber <f.weber@proxmox.com>
> Suggested-by: Christian Ebner <c.ebner@proxmox.com>
> Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
> ---
> 
> v2, thanks @Christian:
>  - Removed unneeded pw.try_clone()
> 
>  proxmox-backup-client/src/mount.rs | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/proxmox-backup-client/src/mount.rs b/proxmox-backup-client/src/mount.rs
> index b69e7e80..e7718b84 100644
> --- a/proxmox-backup-client/src/mount.rs
> +++ b/proxmox-backup-client/src/mount.rs
> @@ -1,6 +1,7 @@
>  use std::collections::HashMap;
>  use std::ffi::OsStr;
>  use std::hash::BuildHasher;
> +use std::io::{self, Write};
>  use std::os::unix::io::{AsRawFd, OwnedFd};
>  use std::path::{Path, PathBuf};
>  use std::sync::Arc;
> @@ -189,7 +190,10 @@ fn mount(
>          Ok(ForkResult::Child) => {
>              drop(pr);
>              nix::unistd::setsid().unwrap();
> -            proxmox_async::runtime::main(mount_do(param, Some(pw)))
> +            let result = proxmox_async::runtime::main(mount_do(param, Some(pw)));
> +            io::stdout().flush()?;
> +            io::stderr().flush()?;
> +            result
>          }
>          Err(_) => bail!("failed to daemonize process"),
>      }


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Christian Ebner]

On 6/18/24 13:20, Friedrich Weber wrote:
> Thanks for tackling this! I ran into this issue (sometimes missing error
> output) when I was running `sudo proxmox-backup-client mount [...]` and
> forgot that PBS_REPOSITORY was not passed through by sudo. It looks like
> in this case, the patch doesn't completely fix this issue -- seems like
> I'm still missing the error output sometimes:
> 
> $ sudo ./proxmox-backup-client mount \
>          "$SNAPSHOT" $FILE $PATH \
>          --keyfile $KEYFILE --ns $NS
> $ sudo ./proxmox-backup-client mount \
>          "$SNAPSHOT" $FILE $PATH \
>          --keyfile $KEYFILE --ns $NS
> $ sudo ./proxmox-backup-client mount \
>          "$SNAPSHOT" $FILE $PATH \
>          --keyfile $KEYFILE --ns $NS
> Error: unable to get (default) repository

Hi Friedrich,
I just double checked by invoking the command in a for loop 10 times, 
for all of which I do get the error message as output.

```bash
for i in $(seq 0 9)
do
	sudo proxmox-backup-client mount ct/100/2024-06-13T10:38:54Z root.pxar /mnt
done
```

Interestingly, I do also get the output without the patch applied, so 
therefore I did not notice (and obviously forgot to do the negative test 
during review).

Maybe some difference in setup?

Cheers,
Chris


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Gabriel Goller]

On 18.06.2024 13:41, Christian Ebner wrote:
>On 6/18/24 13:20, Friedrich Weber wrote:
>>Thanks for tackling this! I ran into this issue (sometimes missing error
>>output) when I was running `sudo proxmox-backup-client mount [...]` and
>>forgot that PBS_REPOSITORY was not passed through by sudo. It looks like
>>in this case, the patch doesn't completely fix this issue -- seems like
>>I'm still missing the error output sometimes:
>>
>>$ sudo ./proxmox-backup-client mount \
>>         "$SNAPSHOT" $FILE $PATH \
>>         --keyfile $KEYFILE --ns $NS
>>$ sudo ./proxmox-backup-client mount \
>>         "$SNAPSHOT" $FILE $PATH \
>>         --keyfile $KEYFILE --ns $NS
>>$ sudo ./proxmox-backup-client mount \
>>         "$SNAPSHOT" $FILE $PATH \
>>         --keyfile $KEYFILE --ns $NS
>>Error: unable to get (default) repository
>
>Hi Friedrich,
>I just double checked by invoking the command in a for loop 10 times, 
>for all of which I do get the error message as output.
>
>```bash
>for i in $(seq 0 9)
>do
>	sudo proxmox-backup-client mount ct/100/2024-06-13T10:38:54Z root.pxar /mnt
>done
>```
>
>Interestingly, I do also get the output without the patch applied, so 
>therefore I did not notice (and obviously forgot to do the negative 
>test during review).
>
>Maybe some difference in setup?
>
>Cheers,
>Chris

Hmm I think I know what happens...
This patch is flaky for me as well, sometimes it works, sometimes it
doesn't. The problem is that we move the `pw` file descriptor to
`mount_do`, which means that after it returns, the file_descriptor gets
dropped.

     let (pr, pw) = proxmox_sys::pipe()?;
     let pr: OwnedFd = pr.into(); // until next sys bump
     let pw: OwnedFd = pw.into();
     match unsafe { fork() } {
         Ok(ForkResult::Parent { .. }) => {
             drop(pw);
             // Blocks the parent process until we are ready to go in the child
             let _res = nix::unistd::read(pr.as_raw_fd(), &mut [0]).unwrap();
             Ok(Value::Null)
         }
         Ok(ForkResult::Child) => {
             drop(pr);
             nix::unistd::setsid().unwrap();
             let result = proxmox_async::runtime::main(mount_do(param, Some(pw)));
                                                                            ^^^
pw gets dropped here, which means the fd is closed.

             io::stdout().flush()?;
             io::stderr().flush()?;
             result
         }
         Err(_) => bail!("failed to daemonize process"),
     }


In the parent process we read from the fd—which is then closed—meaning
that the parent exits at the same time as we do the stderr/stdout flush.

And actually this patch was stupid to begin with, because we don't even
print the error in the mound_do function, but later in the api-handler
:(.

I think we need to do something like this:

     if let Err(e) = nix::unistd::read(pr.as_raw_fd(), &mut [0]) {
         // if the fd is closed, or some other error happens, wait so
         // that the child can return the error safely, then exit 
         nix::sys::wait::wait().unwrap();
     }
     Ok(Value::Null)




_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Friedrich Weber]

Hi Chris,

On 18/06/2024 13:41, Christian Ebner wrote:
> On 6/18/24 13:20, Friedrich Weber wrote:
>> Thanks for tackling this! I ran into this issue (sometimes missing error
>> output) when I was running `sudo proxmox-backup-client mount [...]` and
>> forgot that PBS_REPOSITORY was not passed through by sudo. It looks like
>> in this case, the patch doesn't completely fix this issue -- seems like
>> I'm still missing the error output sometimes:
>>
>> $ sudo ./proxmox-backup-client mount \
>>          "$SNAPSHOT" $FILE $PATH \
>>          --keyfile $KEYFILE --ns $NS
>> $ sudo ./proxmox-backup-client mount \
>>          "$SNAPSHOT" $FILE $PATH \
>>          --keyfile $KEYFILE --ns $NS
>> $ sudo ./proxmox-backup-client mount \
>>          "$SNAPSHOT" $FILE $PATH \
>>          --keyfile $KEYFILE --ns $NS
>> Error: unable to get (default) repository
> 
> Hi Friedrich,
> I just double checked by invoking the command in a for loop 10 times,
> for all of which I do get the error message as output.
> 
> ```bash
> for i in $(seq 0 9)
> do
>     sudo proxmox-backup-client mount ct/100/2024-06-13T10:38:54Z
> root.pxar /mnt
> done
> ```
> 
> Interestingly, I do also get the output without the patch applied, so
> therefore I did not notice (and obviously forgot to do the negative test
> during review).
> 
> Maybe some difference in setup?

Thanks for double-checking! That's quite weird. Could it be related to
sudo's `use_pty` setting?

With `Defaults use_pty` in /etc/sudoers (the default for me) ...

$ for i in $(seq 10); do sudo ./proxmox-backup-client mount foo bar baz;
done

... gives me only 2-5 lines of output.

If I comment out the `Defaults use_pty` line in /etc/sudoers, I get 10
lines of output consistently.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Christian Ebner]

On 6/18/24 14:32, Friedrich Weber wrote:
> Hi Chris,
> 
> On 18/06/2024 13:41, Christian Ebner wrote:
>> On 6/18/24 13:20, Friedrich Weber wrote:
>>> Thanks for tackling this! I ran into this issue (sometimes missing error
>>> output) when I was running `sudo proxmox-backup-client mount [...]` and
>>> forgot that PBS_REPOSITORY was not passed through by sudo. It looks like
>>> in this case, the patch doesn't completely fix this issue -- seems like
>>> I'm still missing the error output sometimes:
>>>
>>> $ sudo ./proxmox-backup-client mount \
>>>           "$SNAPSHOT" $FILE $PATH \
>>>           --keyfile $KEYFILE --ns $NS
>>> $ sudo ./proxmox-backup-client mount \
>>>           "$SNAPSHOT" $FILE $PATH \
>>>           --keyfile $KEYFILE --ns $NS
>>> $ sudo ./proxmox-backup-client mount \
>>>           "$SNAPSHOT" $FILE $PATH \
>>>           --keyfile $KEYFILE --ns $NS
>>> Error: unable to get (default) repository
>>
>> Hi Friedrich,
>> I just double checked by invoking the command in a for loop 10 times,
>> for all of which I do get the error message as output.
>>
>> ```bash
>> for i in $(seq 0 9)
>> do
>>      sudo proxmox-backup-client mount ct/100/2024-06-13T10:38:54Z
>> root.pxar /mnt
>> done
>> ```
>>
>> Interestingly, I do also get the output without the patch applied, so
>> therefore I did not notice (and obviously forgot to do the negative test
>> during review).
>>
>> Maybe some difference in setup?
> 
> Thanks for double-checking! That's quite weird. Could it be related to
> sudo's `use_pty` setting?

Indeed, I did not have this in my `/etc/sudoers`. Adding it and running 
the for loop again gave me no output at all!

> 
> With `Defaults use_pty` in /etc/sudoers (the default for me) ...
> 
> $ for i in $(seq 10); do sudo ./proxmox-backup-client mount foo bar baz;
> done
> 
> ... gives me only 2-5 lines of output.
> 
> If I comment out the `Defaults use_pty` line in /etc/sudoers, I get 10 > lines of output consistently.

Cheers,
Chris



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Gabriel Goller]

On 18.06.2024 14:32, Friedrich Weber wrote:
>Hi Chris,
>
>On 18/06/2024 13:41, Christian Ebner wrote:
>> On 6/18/24 13:20, Friedrich Weber wrote:
>>> Thanks for tackling this! I ran into this issue (sometimes missing error
>>> output) when I was running `sudo proxmox-backup-client mount [...]` and
>>> forgot that PBS_REPOSITORY was not passed through by sudo. It looks like
>>> in this case, the patch doesn't completely fix this issue -- seems like
>>> I'm still missing the error output sometimes:
>>>
>>> $ sudo ./proxmox-backup-client mount \
>>>          "$SNAPSHOT" $FILE $PATH \
>>>          --keyfile $KEYFILE --ns $NS
>>> $ sudo ./proxmox-backup-client mount \
>>>          "$SNAPSHOT" $FILE $PATH \
>>>          --keyfile $KEYFILE --ns $NS
>>> $ sudo ./proxmox-backup-client mount \
>>>          "$SNAPSHOT" $FILE $PATH \
>>>          --keyfile $KEYFILE --ns $NS
>>> Error: unable to get (default) repository
>>
>> Hi Friedrich,
>> I just double checked by invoking the command in a for loop 10 times,
>> for all of which I do get the error message as output.
>>
>> ```bash
>> for i in $(seq 0 9)
>> do
>>     sudo proxmox-backup-client mount ct/100/2024-06-13T10:38:54Z
>> root.pxar /mnt
>> done
>> ```
>>
>> Interestingly, I do also get the output without the patch applied, so
>> therefore I did not notice (and obviously forgot to do the negative test
>> during review).
>>
>> Maybe some difference in setup?
>
>Thanks for double-checking! That's quite weird. Could it be related to
>sudo's `use_pty` setting?
>
>With `Defaults use_pty` in /etc/sudoers (the default for me) ...
>
>$ for i in $(seq 10); do sudo ./proxmox-backup-client mount foo bar baz;
>done
>
>... gives me only 2-5 lines of output.
>
>If I comment out the `Defaults use_pty` line in /etc/sudoers, I get 10
>lines of output consistently.

This fixes it for me as well when using sudo!

The output without sudo is also somewhat broken though. I always get:

     proxmox-backup-client mount "host/debian/2024-03-11T13:38:32Z" test.pxar ./mnt/
     root@debian:~/cool_mnt# Error: unable to get (default) repository



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Christian Ebner]

On 6/18/24 14:29, Gabriel Goller wrote:
> 
> Hmm I think I know what happens...
> This patch is flaky for me as well, sometimes it works, sometimes it
> doesn't. The problem is that we move the `pw` file descriptor to
> `mount_do`, which means that after it returns, the file_descriptor gets
> dropped.
> 
>      let (pr, pw) = proxmox_sys::pipe()?;
>      let pr: OwnedFd = pr.into(); // until next sys bump
>      let pw: OwnedFd = pw.into();
>      match unsafe { fork() } {
>          Ok(ForkResult::Parent { .. }) => {
>              drop(pw);
>              // Blocks the parent process until we are ready to go in 
> the child
>              let _res = nix::unistd::read(pr.as_raw_fd(), &mut 
> [0]).unwrap();
>              Ok(Value::Null)
>          }
>          Ok(ForkResult::Child) => {
>              drop(pr);
>              nix::unistd::setsid().unwrap();
>              let result = proxmox_async::runtime::main(mount_do(param, 
> Some(pw)));
>                                                                             ^^^
> pw gets dropped here, which means the fd is closed.
> 
>              io::stdout().flush()?;
>              io::stderr().flush()?;
>              result
>          }
>          Err(_) => bail!("failed to daemonize process"),
>      }
> 
> 
> In the parent process we read from the fd—which is then closed—meaning
> that the parent exits at the same time as we do the stderr/stdout flush.
> 
> And actually this patch was stupid to begin with, because we don't even
> print the error in the mound_do function, but later in the api-handler
> :(.

That is actually true, no point in flushing here. So my suggestion for 
the fix was wrong, as I was primed on the error location :(

> 
> I think we need to do something like this:
> 
>      if let Err(e) = nix::unistd::read(pr.as_raw_fd(), &mut [0]) {
>          // if the fd is closed, or some other error happens, wait so
>          // that the child can return the error safely, then exit 
>          nix::sys::wait::wait().unwrap();
>      }
>      Ok(Value::Null)
> 
> 


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup v2] client: mount: flush output before exiting

[Gabriel Goller]

On 18.06.2024 15:00, Christian Ebner wrote:
>On 6/18/24 14:29, Gabriel Goller wrote:
>>
>>Hmm I think I know what happens...
>>This patch is flaky for me as well, sometimes it works, sometimes it
>>doesn't. The problem is that we move the `pw` file descriptor to
>>`mount_do`, which means that after it returns, the file_descriptor gets
>>dropped.
>>
>>     let (pr, pw) = proxmox_sys::pipe()?;
>>     let pr: OwnedFd = pr.into(); // until next sys bump
>>     let pw: OwnedFd = pw.into();
>>     match unsafe { fork() } {
>>         Ok(ForkResult::Parent { .. }) => {
>>             drop(pw);
>>             // Blocks the parent process until we are ready to go 
>>in the child
>>             let _res = nix::unistd::read(pr.as_raw_fd(), &mut 
>>[0]).unwrap();
>>             Ok(Value::Null)
>>         }
>>         Ok(ForkResult::Child) => {
>>             drop(pr);
>>             nix::unistd::setsid().unwrap();
>>             let result = 
>>proxmox_async::runtime::main(mount_do(param, Some(pw)));
>>                                                                            ^^^
>>pw gets dropped here, which means the fd is closed.
>>
>>             io::stdout().flush()?;
>>             io::stderr().flush()?;
>>             result
>>         }
>>         Err(_) => bail!("failed to daemonize process"),
>>     }
>>
>>
>>In the parent process we read from the fd—which is then closed—meaning
>>that the parent exits at the same time as we do the stderr/stdout flush.
>>
>>And actually this patch was stupid to begin with, because we don't even
>>print the error in the mound_do function, but later in the api-handler
>>:(.
>
>That is actually true, no point in flushing here. So my suggestion for 
>the fix was wrong, as I was primed on the error location :(
>

And I blindly followed :)

>>
>>I think we need to do something like this:
>>
>>     if let Err(e) = nix::unistd::read(pr.as_raw_fd(), &mut [0]) {
>>         // if the fd is closed, or some other error happens, wait so
>>         // that the child can return the error safely, then exit  
>>        nix::sys::wait::wait().unwrap();
>>     }
>>     Ok(Value::Null)

We actually need to also check the Ok branch if the passed buffer is
empty because if the fd get's closed we don't get an error code back.

Will submit a v3 shortly!


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel