How to deal with missing AppArmor profiles for microk8s on LXD

Today I will describe how to deal with missing AppArmor profiles for microk8s on LXD.

Initial information

Guest operating system version.

$ lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

LXD version on the host operating system.

$ lxd --version

4.0.2

Install apparmor-utils inside the guest operating system.

$ sudo apt install apparmor-utils

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  python3-apparmor python3-libapparmor
Suggested packages:
  vim-addon-manager
The following NEW packages will be installed:
  apparmor-utils python3-apparmor python3-libapparmor
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 157 kB of archives.
After this operation, 966 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-libapparmor amd64 2.13.3-7ubuntu5.1 [26.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-apparmor amd64 2.13.3-7ubuntu5.1 [78.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apparmor-utils amd64 2.13.3-7ubuntu5.1 [51.4 kB]
Fetched 157 kB in 0s (589 kB/s)
Selecting previously unselected package python3-libapparmor.
(Reading database ... 18379 files and directories currently installed.)
Preparing to unpack .../python3-libapparmor_2.13.3-7ubuntu5.1_amd64.deb ...
Unpacking python3-libapparmor (2.13.3-7ubuntu5.1) ...
Selecting previously unselected package python3-apparmor.
Preparing to unpack .../python3-apparmor_2.13.3-7ubuntu5.1_amd64.deb ...
Unpacking python3-apparmor (2.13.3-7ubuntu5.1) ...
Selecting previously unselected package apparmor-utils.
Preparing to unpack .../apparmor-utils_2.13.3-7ubuntu5.1_amd64.deb ...
Unpacking apparmor-utils (2.13.3-7ubuntu5.1) ...
Setting up python3-libapparmor (2.13.3-7ubuntu5.1) ...
Setting up python3-apparmor (2.13.3-7ubuntu5.1) ...
Setting up apparmor-utils (2.13.3-7ubuntu5.1) ...

The issue

microk8s does not start as it cannot change apparmor profile.

$ microk8s

cannot change profile for the next exec call: No such file or directory

Guest logs will indicate the same issue.

$ sudo tail /var/log/kern.log

[...]
Jul  4 20:51:55 kube-worker-3 kernel: [   40.914921] audit: type=1400 audit(1593895915.057:50): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/snap/core/9436/usr/lib/snapd/snap-confine" name="snap.microk8s.daemon-containerd" pid=6308 comm="snap-confine"
Jul  4 20:51:55 kube-worker-3 kernel: [   40.915857] audit: type=1400 audit(1593895915.057:52): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/snap/core/9436/usr/lib/snapd/snap-confine" name="snap.microk8s.daemon-scheduler" pid=6314 comm="snap-confine"
Jul  4 20:51:55 kube-worker-3 kernel: [   40.917363] audit: type=1400 audit(1593895915.061:54): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/snap/core/9436/usr/lib/snapd/snap-confine" name="snap.microk8s.daemon-proxy" pid=6313 comm="snap-confine"
Jul  4 20:51:55 kube-worker-3 kernel: [   40.917501] audit: type=1400 audit(1593895915.061:55): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/snap/core/9436/usr/lib/snapd/snap-confine" name="snap.microk8s.daemon-apiserver" pid=6306 comm="snap-confine"
Jul  4 20:52:03 kube-worker-3 kernel: [   48.863759] audit: type=1400 audit(1593895923.034:146): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/snap/core/9436/usr/lib/snapd/snap-confine" name="snap.microk8s.daemon-proxy" pid=8382 comm="snap-confine"
Jul  4 20:52:03 kube-worker-3 kernel: [   48.867327] audit: type=1400 audit(1593895923.034:149): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/snap/core/9436/usr/lib/snapd/snap-confine" name="snap.microk8s.daemon-etcd" pid=8379 comm="snap-confine"
Jul  4 20:52:03 kube-worker-3 kernel: [   48.874465] audit: type=1400 audit(1593895923.042:152): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/snap/core/9436/usr/lib/snapd/snap-confine" name="snap.microk8s.daemon-apiserver" pid=8375 comm="snap-confine"
[...]

Issue details

Execute microk8s snap using debug mode to confirm that it cannot change AppArmor profile to snap.microk8s.microk8.

$ SNAPD_DEBUG=1 SNAP_DEBUG_CONFINE=1 microk8s

2020/07/04 21:37:58.884019 cmd_linux.go:207: DEBUG: restarting into "/snap/core/current/usr/bin/snap"
2020/07/04 21:37:58.900289 cmd_run.go:398: DEBUG: SELinux not enabled
DEBUG: umask reset, old umask was   02
DEBUG: security tag: snap.microk8s.microk8s
DEBUG: executable:   /snap/core/9665/usr/lib/snapd/snap-exec
DEBUG: confinement:  classic
DEBUG: base snap:    core
DEBUG: ruid: 2018, euid: 0, suid: 0
DEBUG: rgid: 2018, egid: 2018, sgid: 2018
DEBUG: apparmor label on snap-confine is: /snap/core/9665/usr/lib/snapd/snap-confine
DEBUG: apparmor mode is: enforce
DEBUG: preparing classic execution environment
DEBUG: creating lock directory /run/snapd/lock (if missing)
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: opening lock directory /run/snapd/lock
DEBUG: set_effective_identity uid:0 (change: no), gid:2018 (change: yes)
DEBUG: opening lock file: /run/snapd/lock/microk8s.lock
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: set_effective_identity uid:0 (change: no), gid:2018 (change: yes)
DEBUG: sanity timeout initialized and set for 30 seconds
DEBUG: acquiring exclusive lock (scope microk8s, uid 0)
DEBUG: sanity timeout reset and disabled
DEBUG: releasing lock 5
DEBUG: set_effective_identity uid:2018 (change: yes), gid:2018 (change: yes)
DEBUG: creating user data directory: /home/ansible/snap/microk8s/1496
DEBUG: requesting changing of apparmor profile on next exec to snap.microk8s.microk8s
cannot change profile for the next exec call: No such file or directory

AppArmor is enabled inside the guest operating system.

$ aa-enabled

Yes

Currently loaded AppArmor policy inside the guest operating system.

$ aa-status

apparmor module is loaded.
17 profiles are loaded.
17 profiles are in enforce mode.
   /snap/core/9436/usr/lib/snapd/snap-confine
   /snap/core/9436/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   snap-update-ns.core
   snap-update-ns.lxd
   snap.core.hook.configure
   snap.lxd.activate
   snap.lxd.benchmark
   snap.lxd.buginfo
   snap.lxd.check-kernel
   snap.lxd.daemon
   snap.lxd.hook.configure
   snap.lxd.hook.install
   snap.lxd.hook.remove
   snap.lxd.lxc
   snap.lxd.lxc-to-lxd
   snap.lxd.lxd
   snap.lxd.migrate
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

microk8s profiles are missing as the guest operating system is using unconfined AppArmor profile, so load these by hand to verify that this is the case here.

$ apparmor_parser --add /var/lib/snapd/apparmor/profiles/snap.microk8s.*

AppArmor policy after adding definitions.

$ aa-status

apparmor module is loaded.
53 profiles are loaded.
18 profiles are in enforce mode.
   /snap/core/9436/usr/lib/snapd/snap-confine
   /snap/core/9436/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   snap-update-ns.core
   snap-update-ns.lxd
   snap-update-ns.microk8s
   snap.core.hook.configure
   snap.lxd.activate
   snap.lxd.benchmark
   snap.lxd.buginfo
   snap.lxd.check-kernel
   snap.lxd.daemon
   snap.lxd.hook.configure
   snap.lxd.hook.install
   snap.lxd.hook.remove
   snap.lxd.lxc
   snap.lxd.lxc-to-lxd
   snap.lxd.lxd
   snap.lxd.migrate
35 profiles are in complain mode.
   snap.microk8s.add-node
   snap.microk8s.cilium
   snap.microk8s.config
   snap.microk8s.ctr
   snap.microk8s.daemon-apiserver
   snap.microk8s.daemon-apiserver-kicker
   snap.microk8s.daemon-cluster-agent
   snap.microk8s.daemon-containerd
   snap.microk8s.daemon-controller-manager
   snap.microk8s.daemon-etcd
   snap.microk8s.daemon-flanneld
   snap.microk8s.daemon-kubelet
   snap.microk8s.daemon-proxy
   snap.microk8s.daemon-scheduler
   snap.microk8s.disable
   snap.microk8s.enable
   snap.microk8s.helm
   snap.microk8s.helm3
   snap.microk8s.hook.configure
   snap.microk8s.hook.install
   snap.microk8s.hook.remove
   snap.microk8s.inspect
   snap.microk8s.istioctl
   snap.microk8s.join
   snap.microk8s.juju
   snap.microk8s.kubectl
   snap.microk8s.leave
   snap.microk8s.linkerd
   snap.microk8s.microk8s
   snap.microk8s.refresh-certs
   snap.microk8s.remove-node
   snap.microk8s.reset
   snap.microk8s.start
   snap.microk8s.status
   snap.microk8s.stop
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

microk8s is now usable.

$ microk8s

Available subcommands are:
        add-node
        cilium
        config
        ctr
        disable
        enable
        helm
        helm3
        istioctl
        join
        juju
        kubectl
        leave
        linkerd
        refresh-certs
        remove-node
        reset
        start
        status
        stop
        inspect

It works, so make this change permanent.

First solution

This is the easiest possible solution. Copy microk8s AppArmor profiles to the host operating system.

Create a temporary directory inside the guest operating system.

$ sudo lxc exec kube-worker-3 -- mkdir /tmp/microk8s

Copy microk8s AppArmor profiles.

$ sudo lxc exec kube-worker-3 -- find /var/lib/snapd/apparmor/profiles/ -name "snap.microk8s.*" -exec cp {} /tmp/microk8s/ \;

List files inside the guest operating system.

$ sudo lxc exec kube-worker-3 -- ls /tmp/microk8s/

snap.microk8s.add-node  snap.microk8s.daemon-apiserver         snap.microk8s.daemon-controller-manager  snap.microk8s.daemon-proxy      snap.microk8s.helm            snap.microk8s.hook.remove  snap.microk8s.juju     snap.microk8s.microk8s       snap.microk8s.start
snap.microk8s.cilium    snap.microk8s.daemon-apiserver-kicker  snap.microk8s.daemon-etcd                snap.microk8s.daemon-scheduler  snap.microk8s.helm3           snap.microk8s.inspect      snap.microk8s.kubectl  snap.microk8s.refresh-certs  snap.microk8s.status
snap.microk8s.config    snap.microk8s.daemon-cluster-agent     snap.microk8s.daemon-flanneld            snap.microk8s.disable           snap.microk8s.hook.configure  snap.microk8s.istioctl     snap.microk8s.leave    snap.microk8s.remove-node    snap.microk8s.stop
snap.microk8s.ctr       snap.microk8s.daemon-containerd        snap.microk8s.daemon-kubelet             snap.microk8s.enable            snap.microk8s.hook.install    snap.microk8s.join         snap.microk8s.linkerd  snap.microk8s.reset

Copy these files to the host operating system.

$ sudo lxc file pull -r kube-worker-3/tmp/microk8s /etc/apparmor.d/

List files copied to the host operating system.

$ ls /etc/apparmor.d/microk8s/

snap.microk8s.add-node  snap.microk8s.daemon-apiserver         snap.microk8s.daemon-controller-manager  snap.microk8s.daemon-proxy      snap.microk8s.helm            snap.microk8s.hook.remove  snap.microk8s.juju     snap.microk8s.microk8s       snap.microk8s.start
snap.microk8s.cilium    snap.microk8s.daemon-apiserver-kicker  snap.microk8s.daemon-etcd                snap.microk8s.daemon-scheduler  snap.microk8s.helm3           snap.microk8s.inspect      snap.microk8s.kubectl  snap.microk8s.refresh-certs  snap.microk8s.status
snap.microk8s.config    snap.microk8s.daemon-cluster-agent     snap.microk8s.daemon-flanneld            snap.microk8s.disable           snap.microk8s.hook.configure  snap.microk8s.istioctl     snap.microk8s.leave    snap.microk8s.remove-node    snap.microk8s.stop
snap.microk8s.ctr       snap.microk8s.daemon-containerd        snap.microk8s.daemon-kubelet             snap.microk8s.enable            snap.microk8s.hook.install    snap.microk8s.join         snap.microk8s.linkerd  snap.microk8s.reset

Read microk8s AppArmor profiles inside the host operating system.

$ sudo apparmor_parser --replace /etc/apparmor.d/microk8s/

These profiles will be read automatically after the host operating system reboots.

Second solution

Load microk8s AppArmor profiles inside the guest operating system using rc.local mechanism.

Create /etc/rc.local shell script inside the guest operating system.

$ echo -e '#!/bin/bash\n\napparmor_parser --replace /var/lib/snapd/apparmor/profiles/snap.microk8s.*\nexit 0\n' | sudo tee /etc/rc.local

#!/bin/bash
apparmor_parser --replace /var/lib/snapd/apparmor/profiles/snap.microk8s.*
exit 0

I am using replace action instead of add to avoid unnecessary errors as these profiles will be available system-wide. Yes, it is enough to add additional profiles once.

Ensure that the executable bit is set.

$ sudo chmod +x /etc/rc.local

Execute systemd-rc-local-generator to create a systemd service.

$ /usr/lib/systemd/system-generators/systemd-rc-local-generator

Reboot the guest operating system and inspect rc-local service.

$ systemctl status rc-local

● rc-local.service - /etc/rc.local Compatibility
     Loaded: loaded (/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/rc-local.service.d
             └─debian.conf
     Active: active (exited) since Sat 2020-07-04 22:43:40 UTC; 44s ago
       Docs: man:systemd-rc-local-generator(8)
    Process: 201 ExecStart=/etc/rc.local start (code=exited, status=0/SUCCESS)
Jul 04 22:43:39 kube-master systemd[1]: Starting /etc/rc.local Compatibility...
Jul 04 22:43:40 kube-master systemd[1]: Started /etc/rc.local Compatibility.

microk8s works as expected.

$ microk8s

Available subcommands are:
        add-node
        cilium
        config
	[...]

Initial information#

The issue#

Issue details#

First solution#

Second solution#

Initial information

The issue

Issue details

First solution

Second solution