[pve-devel] [PATCH container] fix #5160: fix move_mount regression for mount point hotplug

[pve-devel] [PATCH container] fix #5160: fix move_mount regression for mount point hotplug

[Filip Schauer]

Set up an Apparmor profile to allow moving mounts for mount point
hotplug.

This fixes a regression caused by
kernel commit 157a3537d6 ("apparmor: Fix regression in mount mediation")

The commit introduced move_mount mediation, which now requires
move_mount to be allowed in the Apparmor profile. Although it is allowed
for most paths in the /usr/bin/lxc-start profile, move_mount is called
with a file descriptor instead of a path in mountpoint_insert_staged,
thus it is not affected by the allow rules in
/etc/apparmor.d/abstractions/lxc/container-base.

To fix this, introduce a new Apparmor profile to allow move_mount on
every mount, specifically for mount point hotplug.

Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
 debian/rules                     | 3 +++
 src/Makefile                     | 3 +++
 src/PVE/LXC.pm                   | 2 +-
 src/pve-container-debug@.service | 1 +
 src/pve-container-mounthotplug   | 7 +++++++
 src/pve-container@.service       | 1 +
 6 files changed, 16 insertions(+), 1 deletion(-)
 create mode 100644 src/pve-container-mounthotplug

diff --git a/debian/rules b/debian/rules
index d999152..f7edccf 100755
--- a/debian/rules
+++ b/debian/rules
@@ -14,3 +14,6 @@
 
 override_dh_installsystemd:
 	dh_installsystemd -ppve-container --no-start --no-enable --no-restart-after-upgrade -r 'system-pve\x2dcontainer.slice'
+
+override_dh_install:
+	dh_apparmor -p pve-container --profile-name=pve-container-mounthotplug
diff --git a/src/Makefile b/src/Makefile
index 5a7a82e..dca666a 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -4,6 +4,7 @@ PREFIX=${DESTDIR}/usr
 BINDIR=${PREFIX}/bin
 LIBDIR=${PREFIX}/lib
 SBINDIR=${PREFIX}/sbin
+ETCDIR=${DESTDIR}/etc
 MANDIR=${PREFIX}/share/man
 DOCDIR=${PREFIX}/share/doc/${PACKAGE}
 LXC_SCRIPT_DIR=${PREFIX}/share/lxc
@@ -13,6 +14,7 @@ LXC_CONFIG_DIR=${LXC_SCRIPT_DIR}/config
 LXC_COMMON_CONFIG_DIR=${LXC_CONFIG_DIR}/common.conf.d
 LXC_USERNS_CONFIG_DIR=${LXC_CONFIG_DIR}/userns.conf.d
 SERVICEDIR=${DESTDIR}/lib/systemd/system
+APPARMORDDIR=${ETCDIR}/apparmor.d
 PODDIR=${DOCDIR}/pod
 MAN1DIR=${MANDIR}/man1/
 MAN5DIR=${MANDIR}/man5/
@@ -73,6 +75,7 @@ install: pct lxc-pve.conf pct.1 pct.conf.5 pct.bash-completion pct.zsh-completio
 	gzip -9 ${MAN5DIR}/pct.conf.5
 	cd ${MAN5DIR}; ln -s pct.conf.5.gz ct.conf.5.gz
 	install -D -m 0644 10-pve-ct-inotify-limits.conf ${LIBDIR}/sysctl.d/10-pve-ct-inotify-limits.conf
+	install -D -m 0644 pve-container-mounthotplug ${APPARMORDDIR}/pve/pve-container-mounthotplug
 
 pve-userns.seccomp: /usr/share/lxc/config/common.seccomp
 	cp $< $@
diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
index 7883cfb..7db4833 100644
--- a/src/PVE/LXC.pm
+++ b/src/PVE/LXC.pm
@@ -1974,7 +1974,7 @@ sub mountpoint_hotplug :prototype($$$$$) {
 	my $dir = get_staging_mount_path($opt);
 
 	# Now switch our apparmor profile before mounting:
-	my $data = 'changeprofile /usr/bin/lxc-start';
+	my $data = 'changeprofile pve-container-mounthotplug';
 	if (syswrite($aa_fd, $data, length($data)) != length($data)) {
 	    die "failed to change apparmor profile: $!\n";
 	}
diff --git a/src/pve-container-debug@.service b/src/pve-container-debug@.service
index 7cfebaa..66b5d9f 100644
--- a/src/pve-container-debug@.service
+++ b/src/pve-container-debug@.service
@@ -13,6 +13,7 @@ Type=simple
 Delegate=yes
 KillMode=mixed
 TimeoutStopSec=120s
+ExecStartPre=/lib/apparmor/profile-load pve/pve-container-mounthotplug
 ExecStart=/usr/bin/lxc-start -F -n %i -o /dev/stderr -l DEBUG
 ExecStop=/usr/share/lxc/pve-container-stop-wrapper %i
 # Environment=BOOTUP=serial
diff --git a/src/pve-container-mounthotplug b/src/pve-container-mounthotplug
new file mode 100644
index 0000000..e6f3903
--- /dev/null
+++ b/src/pve-container-mounthotplug
@@ -0,0 +1,7 @@
+#include <tunables/global>
+
+profile pve-container-mounthotplug flags=(attach_disconnected) {
+  #include <abstractions/lxc/start-container>
+
+  mount options=(move),
+}
diff --git a/src/pve-container@.service b/src/pve-container@.service
index fdc373e..011565b 100644
--- a/src/pve-container@.service
+++ b/src/pve-container@.service
@@ -13,6 +13,7 @@ Type=simple
 Delegate=yes
 KillMode=mixed
 TimeoutStopSec=120s
+ExecStartPre=/lib/apparmor/profile-load pve/pve-container-mounthotplug
 ExecStart=/usr/bin/lxc-start -F -n %i
 ExecStop=/usr/share/lxc/pve-container-stop-wrapper %i
 # Environment=BOOTUP=serial
-- 
2.39.2

Re: [pve-devel] [PATCH container] fix #5160: fix move_mount regression for mount point hotplug

[Fiona Ebner]

Am 08.01.24 um 14:54 schrieb Filip Schauer:
> Set up an Apparmor profile to allow moving mounts for mount point
> hotplug.
> 
> This fixes a regression caused by
> kernel commit 157a3537d6 ("apparmor: Fix regression in mount mediation")
> 
> The commit introduced move_mount mediation, which now requires
> move_mount to be allowed in the Apparmor profile. Although it is allowed
> for most paths in the /usr/bin/lxc-start profile, move_mount is called
> with a file descriptor instead of a path in mountpoint_insert_staged,
> thus it is not affected by the allow rules in
> /etc/apparmor.d/abstractions/lxc/container-base.
> 

Would it be difficult/impossible for us to do it path-based instead?

> To fix this, introduce a new Apparmor profile to allow move_mount on
> every mount, specifically for mount point hotplug.
> 
> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>

Tested-by: Fiona Ebner <f.ebner@proxmox.com>

Fixes the issue and makes hotplug work again :)

However, I did run into one minor thing (number 1 below), but not sure
if it can be improved. Except the part about the undef warning which is
pre-existing. Another pre-existing issue I ran into is number 2 below,
should also be investigated, but is not caused by this patch.

1.

For an already running container, I got

> mp0: unable to hotplug mp0: failed to change apparmor profile:

after installing the patched pve-container package and attempting to
hotplug. Maybe that's expected and no big deal, because hotplug already
didn't work.

Checking in the journal, it's:

> AVC apparmor="DENIED" operation="change_profile" class="file" info="label not found" error=-2 profile="unconfined" name="pve-container-mounthotplug" pid=99919 comm=7076656461656D6F6E20776F726B65
> Use of uninitialized value in numeric ne (!=) at /usr/share/perl5/PVE/LXC.pm line 1978.

The latter warning seems to be pre-existing for the syswrite comparison
below:

> @@ -1974,7 +1974,7 @@ sub mountpoint_hotplug :prototype($$$$$) {
>  	my $dir = get_staging_mount_path($opt);
>  
>  	# Now switch our apparmor profile before mounting:
> -	my $data = 'changeprofile /usr/bin/lxc-start';
> +	my $data = 'changeprofile pve-container-mounthotplug';
>  	if (syswrite($aa_fd, $data, length($data)) != length($data)) {
>  	    die "failed to change apparmor profile: $!\n";
>  	}


But then $! should be set according to 'perldoc -f syswrite' which it
wasn't telling from my quoted error message above  ¯\_(ツ)_/¯

> Returns the number of bytes actually
>             written, or "undef" if there was an error (in this case the
>             errno variable $! is also set)

2.

After shutdown+start, the hotplug worked as expected. Played around a
bit more and something strange happened with LVM as the storage: While
it worked the first time, when I shut down the container, detached the
mount point, started again and then tried to hot-plug, I got:
mp1: unable to hotplug mp1: command 'mount /dev/lvm/vm-129-disk-1
/var/lib/lxc/.pve-staged-mounts/mp1' failed: exit code 32

Doesn't seem to happen with ZFS or directory storage and starting with
the mount point pre-attached also worked fine with previously written
files still there.

Also booted into 6.2.16-20-pve, installed unpatched pve-container and it
seems the issue was already present then.

Re: [pve-devel] [PATCH container] fix #5160: fix move_mount regression for mount point hotplug

[Fiona Ebner]

Am 25.03.24 um 11:29 schrieb Fiona Ebner:
> 
> After shutdown+start, the hotplug worked as expected. Played around a
> bit more and something strange happened with LVM as the storage: While
> it worked the first time, when I shut down the container, detached the
> mount point, started again and then tried to hot-plug, I got:
> mp1: unable to hotplug mp1: command 'mount /dev/lvm/vm-129-disk-1
> /var/lib/lxc/.pve-staged-mounts/mp1' failed: exit code 32
> 
> Doesn't seem to happen with ZFS or directory storage and starting with
> the mount point pre-attached also worked fine with previously written
> files still there.
> 
> Also booted into 6.2.16-20-pve, installed unpatched pve-container and it
> seems the issue was already present then.
> 

After looking into this, it's likely because the LV is deactivated after
the shutdown. Apparently we're missing the activate_volume() call before
doing a hotplug.

Re: [pve-devel] [PATCH container] fix #5160: fix move_mount regression for mount point hotplug

[Filip Schauer]

On 25/03/2024 11:29, Fiona Ebner wrote:
> Am 08.01.24 um 14:54 schrieb Filip Schauer:
>> Set up an Apparmor profile to allow moving mounts for mount point
>> hotplug.
>>
>> This fixes a regression caused by
>> kernel commit 157a3537d6 ("apparmor: Fix regression in mount mediation")
>>
>> The commit introduced move_mount mediation, which now requires
>> move_mount to be allowed in the Apparmor profile. Although it is allowed
>> for most paths in the /usr/bin/lxc-start profile, move_mount is called
>> with a file descriptor instead of a path in mountpoint_insert_staged,
>> thus it is not affected by the allow rules in
>> /etc/apparmor.d/abstractions/lxc/container-base.
>>
> Would it be difficult/impossible for us to do it path-based instead?


A path-based approach would not work as things are right now, since we
use the file descriptor to issue a move_mount from within the
container's namespace, where we cannot access the staging directory with
a path.

>> To fix this, introduce a new Apparmor profile to allow move_mount on
>> every mount, specifically for mount point hotplug.
>>
>> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
> Tested-by: Fiona Ebner <f.ebner@proxmox.com>
>
> Fixes the issue and makes hotplug work again :)
>
> However, I did run into one minor thing (number 1 below), but not sure
> if it can be improved. Except the part about the undef warning which is
> pre-existing. Another pre-existing issue I ran into is number 2 below,
> should also be investigated, but is not caused by this patch.
>
> 1.
>
> For an already running container, I got
>
>> mp0: unable to hotplug mp0: failed to change apparmor profile:
> after installing the patched pve-container package and attempting to
> hotplug. Maybe that's expected and no big deal, because hotplug already
> didn't work.
>
> Checking in the journal, it's:
>
>> AVC apparmor="DENIED" operation="change_profile" class="file" info="label not found" error=-2 profile="unconfined" name="pve-container-mounthotplug" pid=99919 comm=7076656461656D6F6E20776F726B65


This happens because the new apparmor profile
"pve-container-mounthotplug" is not loaded correctly in postinst.
This is fixed in v2.


>> Use of uninitialized value in numeric ne (!=) at /usr/share/perl5/PVE/LXC.pm line 1978.
> The latter warning seems to be pre-existing for the syswrite comparison
> below:
>
>> @@ -1974,7 +1974,7 @@ sub mountpoint_hotplug :prototype($$$$$) {
>>   	my $dir = get_staging_mount_path($opt);
>>   
>>   	# Now switch our apparmor profile before mounting:
>> -	my $data = 'changeprofile /usr/bin/lxc-start';
>> +	my $data = 'changeprofile pve-container-mounthotplug';
>>   	if (syswrite($aa_fd, $data, length($data)) != length($data)) {
>>   	    die "failed to change apparmor profile: $!\n";
>>   	}
>
> But then $! should be set according to 'perldoc -f syswrite' which it
> wasn't telling from my quoted error message above  ¯\_(ツ)_/¯
>
>> Returns the number of bytes actually
>>              written, or "undef" if there was an error (in this case the
>>              errno variable $! is also set)


A fix for the warning is in the v2.


> 2.
>
> After shutdown+start, the hotplug worked as expected. Played around a
> bit more and something strange happened with LVM as the storage: While
> it worked the first time, when I shut down the container, detached the
> mount point, started again and then tried to hot-plug, I got:
> mp1: unable to hotplug mp1: command 'mount /dev/lvm/vm-129-disk-1
> /var/lib/lxc/.pve-staged-mounts/mp1' failed: exit code 32
>
> Doesn't seem to happen with ZFS or directory storage and starting with
> the mount point pre-attached also worked fine with previously written
> files still there.
>
> Also booted into 6.2.16-20-pve, installed unpatched pve-container and it
> seems the issue was already present then.

Re: [pve-devel] [PATCH container] fix #5160: fix move_mount regression for mount point hotplug

[Filip Schauer]

Patch v2 is available:
https://lists.proxmox.com/pipermail/pve-devel/2024-March/062390.html

On 08/01/2024 14:54, Filip Schauer wrote:
> Set up an Apparmor profile to allow moving mounts for mount point
> hotplug.
>
> This fixes a regression caused by
> kernel commit 157a3537d6 ("apparmor: Fix regression in mount mediation")
>
> The commit introduced move_mount mediation, which now requires
> move_mount to be allowed in the Apparmor profile. Although it is allowed
> for most paths in the /usr/bin/lxc-start profile, move_mount is called
> with a file descriptor instead of a path in mountpoint_insert_staged,
> thus it is not affected by the allow rules in
> /etc/apparmor.d/abstractions/lxc/container-base.
>
> To fix this, introduce a new Apparmor profile to allow move_mount on
> every mount, specifically for mount point hotplug.
>
> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
> ---
>   debian/rules                     | 3 +++
>   src/Makefile                     | 3 +++
>   src/PVE/LXC.pm                   | 2 +-
>   src/pve-container-debug@.service | 1 +
>   src/pve-container-mounthotplug   | 7 +++++++
>   src/pve-container@.service       | 1 +
>   6 files changed, 16 insertions(+), 1 deletion(-)
>   create mode 100644 src/pve-container-mounthotplug
>
> diff --git a/debian/rules b/debian/rules
> index d999152..f7edccf 100755
> --- a/debian/rules
> +++ b/debian/rules
> @@ -14,3 +14,6 @@
>   
>   override_dh_installsystemd:
>   	dh_installsystemd -ppve-container --no-start --no-enable --no-restart-after-upgrade -r 'system-pve\x2dcontainer.slice'
> +
> +override_dh_install:
> +	dh_apparmor -p pve-container --profile-name=pve-container-mounthotplug
> diff --git a/src/Makefile b/src/Makefile
> index 5a7a82e..dca666a 100644
> --- a/src/Makefile
> +++ b/src/Makefile
> @@ -4,6 +4,7 @@ PREFIX=${DESTDIR}/usr
>   BINDIR=${PREFIX}/bin
>   LIBDIR=${PREFIX}/lib
>   SBINDIR=${PREFIX}/sbin
> +ETCDIR=${DESTDIR}/etc
>   MANDIR=${PREFIX}/share/man
>   DOCDIR=${PREFIX}/share/doc/${PACKAGE}
>   LXC_SCRIPT_DIR=${PREFIX}/share/lxc
> @@ -13,6 +14,7 @@ LXC_CONFIG_DIR=${LXC_SCRIPT_DIR}/config
>   LXC_COMMON_CONFIG_DIR=${LXC_CONFIG_DIR}/common.conf.d
>   LXC_USERNS_CONFIG_DIR=${LXC_CONFIG_DIR}/userns.conf.d
>   SERVICEDIR=${DESTDIR}/lib/systemd/system
> +APPARMORDDIR=${ETCDIR}/apparmor.d
>   PODDIR=${DOCDIR}/pod
>   MAN1DIR=${MANDIR}/man1/
>   MAN5DIR=${MANDIR}/man5/
> @@ -73,6 +75,7 @@ install: pct lxc-pve.conf pct.1 pct.conf.5 pct.bash-completion pct.zsh-completio
>   	gzip -9 ${MAN5DIR}/pct.conf.5
>   	cd ${MAN5DIR}; ln -s pct.conf.5.gz ct.conf.5.gz
>   	install -D -m 0644 10-pve-ct-inotify-limits.conf ${LIBDIR}/sysctl.d/10-pve-ct-inotify-limits.conf
> +	install -D -m 0644 pve-container-mounthotplug ${APPARMORDDIR}/pve/pve-container-mounthotplug
>   
>   pve-userns.seccomp: /usr/share/lxc/config/common.seccomp
>   	cp $< $@
> diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
> index 7883cfb..7db4833 100644
> --- a/src/PVE/LXC.pm
> +++ b/src/PVE/LXC.pm
> @@ -1974,7 +1974,7 @@ sub mountpoint_hotplug :prototype($$$$$) {
>   	my $dir = get_staging_mount_path($opt);
>   
>   	# Now switch our apparmor profile before mounting:
> -	my $data = 'changeprofile /usr/bin/lxc-start';
> +	my $data = 'changeprofile pve-container-mounthotplug';
>   	if (syswrite($aa_fd, $data, length($data)) != length($data)) {
>   	    die "failed to change apparmor profile: $!\n";
>   	}
> diff --git a/src/pve-container-debug@.service b/src/pve-container-debug@.service
> index 7cfebaa..66b5d9f 100644
> --- a/src/pve-container-debug@.service
> +++ b/src/pve-container-debug@.service
> @@ -13,6 +13,7 @@ Type=simple
>   Delegate=yes
>   KillMode=mixed
>   TimeoutStopSec=120s
> +ExecStartPre=/lib/apparmor/profile-load pve/pve-container-mounthotplug
>   ExecStart=/usr/bin/lxc-start -F -n %i -o /dev/stderr -l DEBUG
>   ExecStop=/usr/share/lxc/pve-container-stop-wrapper %i
>   # Environment=BOOTUP=serial
> diff --git a/src/pve-container-mounthotplug b/src/pve-container-mounthotplug
> new file mode 100644
> index 0000000..e6f3903
> --- /dev/null
> +++ b/src/pve-container-mounthotplug
> @@ -0,0 +1,7 @@
> +#include <tunables/global>
> +
> +profile pve-container-mounthotplug flags=(attach_disconnected) {
> +  #include <abstractions/lxc/start-container>
> +
> +  mount options=(move),
> +}
> diff --git a/src/pve-container@.service b/src/pve-container@.service
> index fdc373e..011565b 100644
> --- a/src/pve-container@.service
> +++ b/src/pve-container@.service
> @@ -13,6 +13,7 @@ Type=simple
>   Delegate=yes
>   KillMode=mixed
>   TimeoutStopSec=120s
> +ExecStartPre=/lib/apparmor/profile-load pve/pve-container-mounthotplug
>   ExecStart=/usr/bin/lxc-start -F -n %i
>   ExecStop=/usr/share/lxc/pve-container-stop-wrapper %i
>   # Environment=BOOTUP=serial