[pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[Dominik Csapak]

In general we want all open files to have set CLOEXEC since our
reloading mechanism can basically fork at any moment and we don't want
newer daemons to carry around old file descriptors, especially lock
files.

Since `make_tmp_file` is called by many things (e.g. open_file_locked,
logrotate, rrd), set FD_CLOEXEC after getting the filehandle.

This fixes an issue with e.g. tape backups not working because of such
lingering lock files after a reload.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
there are other code parts where we open file without CLOEXEC, but
wanted to send this for now.

 proxmox-sys/src/fs/file.rs | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
index fbfc0b58..05d0aff0 100644
--- a/proxmox-sys/src/fs/file.rs
+++ b/proxmox-sys/src/fs/file.rs
@@ -7,7 +7,7 @@ use std::time::Duration;
 
 use anyhow::{bail, format_err, Context as _, Error};
 use nix::errno::Errno;
-use nix::fcntl::OFlag;
+use nix::fcntl::{FcntlArg, FdFlag, OFlag};
 use nix::sys::stat;
 use nix::unistd;
 use nix::NixPath;
@@ -128,7 +128,10 @@ pub fn make_tmp_file<P: AsRef<Path>>(
     let mut template = path.to_owned();
     template.set_extension("tmp_XXXXXX");
     let (mut file, tmp_path) = match unistd::mkstemp(&template) {
-        Ok((fd, path)) => (unsafe { File::from_raw_fd(fd) }, path),
+        Ok((fd, path)) => {
+            nix::fcntl::fcntl(fd, FcntlArg::F_SETFD(FdFlag::FD_CLOEXEC))?;
+            (unsafe { File::from_raw_fd(fd) }, path)
+        }
         Err(err) => bail!("mkstemp {:?} failed: {}", template, err),
     };
 
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[Fabian Grünbichler]

> Dominik Csapak <d.csapak@proxmox.com> hat am 28.11.2024 15:54 CET geschrieben:
> In general we want all open files to have set CLOEXEC since our
> reloading mechanism can basically fork at any moment and we don't want
> newer daemons to carry around old file descriptors, especially lock
> files.
> 
> Since `make_tmp_file` is called by many things (e.g. open_file_locked,
> logrotate, rrd), set FD_CLOEXEC after getting the filehandle.
> 
> This fixes an issue with e.g. tape backups not working because of such
> lingering lock files after a reload.

and also one that "leaked" an additional FD for every proxmox-backup-proxy reload via the RRD journal files - so this fixes a bug where PBS will eventually run into the open file limits if you keep reloading that service without ever stopping or restarting it.

might be a good addition to the commit message :)

> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> there are other code parts where we open file without CLOEXEC, but
> wanted to send this for now.
> 
>  proxmox-sys/src/fs/file.rs | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
> index fbfc0b58..05d0aff0 100644
> --- a/proxmox-sys/src/fs/file.rs
> +++ b/proxmox-sys/src/fs/file.rs
> @@ -7,7 +7,7 @@ use std::time::Duration;
>  
>  use anyhow::{bail, format_err, Context as _, Error};
>  use nix::errno::Errno;
> -use nix::fcntl::OFlag;
> +use nix::fcntl::{FcntlArg, FdFlag, OFlag};
>  use nix::sys::stat;
>  use nix::unistd;
>  use nix::NixPath;
> @@ -128,7 +128,10 @@ pub fn make_tmp_file<P: AsRef<Path>>(
>      let mut template = path.to_owned();
>      template.set_extension("tmp_XXXXXX");
>      let (mut file, tmp_path) = match unistd::mkstemp(&template) {
> -        Ok((fd, path)) => (unsafe { File::from_raw_fd(fd) }, path),
> +        Ok((fd, path)) => {
> +            nix::fcntl::fcntl(fd, FcntlArg::F_SETFD(FdFlag::FD_CLOEXEC))?;
> +            (unsafe { File::from_raw_fd(fd) }, path)
> +        }

unfortunately, this is still racy since the FD is open with O_CLOEXEC between the unistd::mkstemp and the fcntl - see the man page of fcntl which explicitly calls this out:

"In  multithreaded programs, using fcntl() F_SETFD to set the close-on-exec flag at the same time as another thread performs a fork(2) plus execve(2) is vulnerable to a race condition that may unintentionally leak the file descriptor to the program executed in the child process."

we could use libc::mkostemp (unsafe, path/template+flags -> raw fd or error as c_int) instead? and/or we could write a wrapper around it and propose it upstream for nix inclusion? ;) but since this seems to be the only place where we call mkstemp..

>          Err(err) => bail!("mkstemp {:?} failed: {}", template, err),
>      };
>  
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[Dominik Csapak]

On 11/29/24 08:57, Fabian Grünbichler wrote:
>> Dominik Csapak <d.csapak@proxmox.com> hat am 28.11.2024 15:54 CET geschrieben:
>> In general we want all open files to have set CLOEXEC since our
>> reloading mechanism can basically fork at any moment and we don't want
>> newer daemons to carry around old file descriptors, especially lock
>> files.
>>
>> Since `make_tmp_file` is called by many things (e.g. open_file_locked,
>> logrotate, rrd), set FD_CLOEXEC after getting the filehandle.
>>
>> This fixes an issue with e.g. tape backups not working because of such
>> lingering lock files after a reload.
> 
> and also one that "leaked" an additional FD for every proxmox-backup-proxy reload via the RRD journal files - so this fixes a bug where PBS will eventually run into the open file limits if you keep reloading that service without ever stopping or restarting it.

nice! did not have too much time yesterday to look into other benefits ;)

> 
> might be a good addition to the commit message :)
> 
>> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
>> ---
>> there are other code parts where we open file without CLOEXEC, but
>> wanted to send this for now.
>>
>>   proxmox-sys/src/fs/file.rs | 7 +++++--
>>   1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
>> index fbfc0b58..05d0aff0 100644
>> --- a/proxmox-sys/src/fs/file.rs
>> +++ b/proxmox-sys/src/fs/file.rs
>> @@ -7,7 +7,7 @@ use std::time::Duration;
>>   
>>   use anyhow::{bail, format_err, Context as _, Error};
>>   use nix::errno::Errno;
>> -use nix::fcntl::OFlag;
>> +use nix::fcntl::{FcntlArg, FdFlag, OFlag};
>>   use nix::sys::stat;
>>   use nix::unistd;
>>   use nix::NixPath;
>> @@ -128,7 +128,10 @@ pub fn make_tmp_file<P: AsRef<Path>>(
>>       let mut template = path.to_owned();
>>       template.set_extension("tmp_XXXXXX");
>>       let (mut file, tmp_path) = match unistd::mkstemp(&template) {
>> -        Ok((fd, path)) => (unsafe { File::from_raw_fd(fd) }, path),
>> +        Ok((fd, path)) => {
>> +            nix::fcntl::fcntl(fd, FcntlArg::F_SETFD(FdFlag::FD_CLOEXEC))?;
>> +            (unsafe { File::from_raw_fd(fd) }, path)
>> +        }
> 
> unfortunately, this is still racy since the FD is open with O_CLOEXEC between the unistd::mkstemp and the fcntl - see the man page of fcntl which explicitly calls this out:
> 
> "In  multithreaded programs, using fcntl() F_SETFD to set the close-on-exec flag at the same time as another thread performs a fork(2) plus execve(2) is vulnerable to a race condition that may unintentionally leak the file descriptor to the program executed in the child process."
> 
> we could use libc::mkostemp (unsafe, path/template+flags -> raw fd or error as c_int) instead? and/or we could write a wrapper around it and propose it upstream for nix inclusion? ;) but since this seems to be the only place where we call mkstemp..
> 

yeah had the same though, and have a version here where i basically copied nix's mkstemp but with 
oflags + mkostemp call to libc, so i can send that if wanted

the question for me is if it's ok to use since mkostemp is only a glibc extension (since 2.7) and we
may use that in proxmox-backup-client which we want to statically build ?
(not sure how that static compilation works with such a thing though...)

>>           Err(err) => bail!("mkstemp {:?} failed: {}", template, err),
>>       };
>>   
>> -- 
>> 2.39.5
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[Fabian Grünbichler]

> Dominik Csapak <d.csapak@proxmox.com> hat am 29.11.2024 09:02 CET geschrieben:
> On 11/29/24 08:57, Fabian Grünbichler wrote:
> >> Dominik Csapak <d.csapak@proxmox.com> hat am 28.11.2024 15:54 CET geschrieben:
> >> @@ -128,7 +128,10 @@ pub fn make_tmp_file<P: AsRef<Path>>(
> >>       let mut template = path.to_owned();
> >>       template.set_extension("tmp_XXXXXX");
> >>       let (mut file, tmp_path) = match unistd::mkstemp(&template) {
> >> -        Ok((fd, path)) => (unsafe { File::from_raw_fd(fd) }, path),
> >> +        Ok((fd, path)) => {
> >> +            nix::fcntl::fcntl(fd, FcntlArg::F_SETFD(FdFlag::FD_CLOEXEC))?;
> >> +            (unsafe { File::from_raw_fd(fd) }, path)
> >> +        }
> > 
> > unfortunately, this is still racy since the FD is open with O_CLOEXEC between the unistd::mkstemp and the fcntl - see the man page of fcntl which explicitly calls this out:
> > 
> > "In  multithreaded programs, using fcntl() F_SETFD to set the close-on-exec flag at the same time as another thread performs a fork(2) plus execve(2) is vulnerable to a race condition that may unintentionally leak the file descriptor to the program executed in the child process."
> > 
> > we could use libc::mkostemp (unsafe, path/template+flags -> raw fd or error as c_int) instead? and/or we could write a wrapper around it and propose it upstream for nix inclusion? ;) but since this seems to be the only place where we call mkstemp..
> > 
> 
> yeah had the same though, and have a version here where i basically copied nix's mkstemp but with 
> oflags + mkostemp call to libc, so i can send that if wanted
> 
> the question for me is if it's ok to use since mkostemp is only a glibc extension (since 2.7) and we
> may use that in proxmox-backup-client which we want to statically build ?
> (not sure how that static compilation works with such a thing though...)

that's a fair argument, if we want that it looks like we'd have to reimplement mkostemp ourselves I guess.. or we abandon this approach and evaluate allow-listing FDs we want to keep, and closing all others on fork..  but then we have to ensure no forking ever happens outside of our immediate control, which seems kinda annoying as well..


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[Thomas Lamprecht]

Am 29.11.24 um 09:02 schrieb Dominik Csapak:
> the question for me is if it's ok to use since mkostemp is only a glibc extension (since 2.7) and we
> may use that in proxmox-backup-client which we want to statically build ?
> (not sure how that static compilation works with such a thing though...)

FWIW, it's supported by MUSL:

http://git.musl-libc.org/cgit/musl/tree/src/temp/mkostemp.c


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[Thomas Lamprecht]

Am 29.11.24 um 11:21 schrieb Thomas Lamprecht:
> Am 29.11.24 um 09:02 schrieb Dominik Csapak:
>> the question for me is if it's ok to use since mkostemp is only a glibc extension (since 2.7) and we
>> may use that in proxmox-backup-client which we want to statically build ?
>> (not sure how that static compilation works with such a thing though...)
> 
> FWIW, it's supported by MUSL:
> 
> http://git.musl-libc.org/cgit/musl/tree/src/temp/mkostemp.c

And just to clarify, the rust libc crate supports different libc targets and
exposes mkostemp among others (all?) for both, the GNU libc a musl libc
targets.

So using that method might be indeed the best option for now and should not
hinder any static builds, as if not with glibc then we highly probably do
them with musl.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox] sys: fs: set FD_CLOEXEC when creating temp files

[Dominik Csapak]

On 11/29/24 13:31, Thomas Lamprecht wrote:
> Am 29.11.24 um 11:21 schrieb Thomas Lamprecht:
>> Am 29.11.24 um 09:02 schrieb Dominik Csapak:
>>> the question for me is if it's ok to use since mkostemp is only a glibc extension (since 2.7) and we
>>> may use that in proxmox-backup-client which we want to statically build ?
>>> (not sure how that static compilation works with such a thing though...)
>>
>> FWIW, it's supported by MUSL:
>>
>> http://git.musl-libc.org/cgit/musl/tree/src/temp/mkostemp.c
> 
> And just to clarify, the rust libc crate supports different libc targets and
> exposes mkostemp among others (all?) for both, the GNU libc a musl libc
> targets.
> 
> So using that method might be indeed the best option for now and should not
> hinder any static builds, as if not with glibc then we highly probably do
> them with musl.

great, thanks for looking that up. I'll send a v2 with using mkostemp
(with a small nix like wrapper around, so we can it as a drop in replacement,
and maybe send it upstream?)




_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel