Alpine Linux package trigger scripts failing on CRI-O

2020-11-11 22:30:00 UTC

I recently created a self-managed Kubernetes cluster using the CRI-O container runtime. I had an issue with Alpine Linux images where package installation trigger scripts were failing for unknown reason (script exited with error 127). Since I couldn’t find a resource online to fix the issue, I wanted to make a blog post to help others that would struggle with it.

Symptom

When installing packages on Alpine Linux, trigger scripts are failing, although installation seems to succeed (exit code being 0):

/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20191127-r4)
(2/4) Installing nghttp2-libs (1.41.0-r0)
(3/4) Installing libcurl (7.69.1-r1)
(4/4) Installing curl (7.69.1-r1)
Executing busybox-1.31.1-r19.trigger
ERROR: busybox-1.31.1-r19.trigger: script exited with error 127
Executing ca-certificates-20191127-r4.trigger
ERROR: ca-certificates-20191127-r4.trigger: script exited with error 127
OK: 7 MiB in 18 packages
/ # echo $?
0

Investigation

I made a custom Alpine Linux image with strace installed (built locally, using Docker). This tool prints all the system calls made by a process.

FROM alpine
RUN apk add --no-cache strace

Then, I’ve run the apk command again using strace (-f argument tracks child processes):

strace -f apk add curl 2>&1 | less

I looked for a trigger script execution in the output and compared with the same command ran locally. On CRI-O, it failed to run the chroot system call:

[pid    17] chroot(".")                 = -1 EPERM (Operation not permitted)

It appears that CRI-O doesn’t allow chroot by default.

Cure

To enable chroot by default on CRI-O, add SYS_CHROOT in the default_capabilities array in /etc/crio/crio.conf:

default_capabilities = [
	# ...
	"SYS_CHROOT",
]

Then, restart CRI-O:

systemctl restart crio

Package installations should work now:

/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20191127-r4)
(2/4) Installing nghttp2-libs (1.41.0-r0)
(3/4) Installing libcurl (7.69.1-r1)
(4/4) Installing curl (7.69.1-r1)
Executing busybox-1.31.1-r19.trigger
Executing ca-certificates-20191127-r4.trigger
OK: 7 MiB in 18 packages

Latest edition: 2021-07-10 23:11:37 UTC