Error when trying to run privileged docker inside dind-rootless

[mind-ar]

I'm creating a project to manage multiple dev environments using docker (like gitpod, but without k8s), and I want to give users the ability to run docker inside their environment (that runs in docker), at the same time, I want to protect the host as much as possible
The architecture is something like this:

• I searched existing issues before opening this one

Expected behavior

Have nested unprivileged container running inside privileged container, running inside rootless-dind container

Actual behavior

Permission denied in nested privileged container:

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/cpuset/docker: permission denied: unknown.

Steps to reproduce the behavior

1 - run an rootless dind container

docker run -dit --rm --name dind --privileged -v omg-certs:/certs/client --device /dev/net/tun:/dev/net/tun docker:dind-rootless

2 - run a docker client container

docker run -it --rm --link dind:docker -v omg-certs:/certs/client  alpine sh
/ # apk add --no-cache docker

3 - run a privileged container inside dind

DOCKER_TLS=1 \
DOCKER_TLS_VERIFY=1 \
DOCKER_HOST=tcp://docker:2376 \
DOCKER_CERT_PATH=/certs/client \
docker run -it --privileged --net=host alpine sh

4 - run docker inside privileged container

apk add --no-cache docker
dockerd &

docker run --rm hello-world

Workaround

1 - init a shell in dind-docker as root

docker exec -it dind -u root sh

2 - create cgroup folders for docker

mkdir -p /sys/fs/cgroup/cpuset/docker
chmod a+rw /sys/fs/cgroup/cpuset/docker/
mkdir -p /sys/fs/cgroup/devices/docker
chmod a+rw /sys/fs/cgroup/devices/docker/
mkdir -p /sys/fs/cgroup/memory/docker
chmod a+rw /sys/fs/cgroup/memory/docker/
mkdir -p /sys/fs/cgroup/pids/docker
chmod a+rw /sys/fs/cgroup/pids/docker/
mkdir -p /sys/fs/cgroup/blkio/docker
chmod a+rw /sys/fs/cgroup/blkio/docker
mkdir -p /sys/fs/cgroup/hugetlb/docker
chmod a+rw /sys/fs/cgroup/hugetlb/docker
mkdir -p /sys/fs/cgroup/perf_event/docker
chmod a+rw /sys/fs/cgroup/perf_event/docker
mkdir -p /sys/fs/cgroup/freezer/docker
chmod a+rw /sys/fs/cgroup/freezer/docker

Additional environment details (AWS, VirtualBox, physical, etc.)

[AkihiroSuda]

Does dockerd --rootless work?

Or does it work on cgroup v2 host? (such as Ubuntu 22.04, Debian 11)

[mind-ar]

Hi, no, that doesn't work.
I did these tests:

Orange and Blue are privileged containers running inside a dind-rootless container

Cases 1 and 2 works when apply the workaround
In case 3, the blue dind-rootless won't start (uid_map failed: operation not permited)

BTW, sorry by my spelling, not a english speaker

[mind-ar]

Updated workaround:

docker-compose exec -u root <container_name> sh -c " \
mkdir -p /sys/fs/cgroup/cpuset/docker && \
mkdir -p /sys/fs/cgroup/devices/docker  && \
mkdir -p /sys/fs/cgroup/memory/docker && \
mkdir -p /sys/fs/cgroup/pids/docker && \
mkdir -p /sys/fs/cgroup/blkio/docker && \
mkdir -p /sys/fs/cgroup/hugetlb/docker && \
mkdir -p /sys/fs/cgroup/perf_event/docker && \
mkdir -p /sys/fs/cgroup/freezer/docker && \
chown rootless /sys/fs/cgroup/cpuset/docker/  && \
chown rootless /sys/fs/cgroup/devices/docker/  && \
chown rootless /sys/fs/cgroup/memory/docker/  && \
chown rootless /sys/fs/cgroup/pids/docker/  && \
chown rootless /sys/fs/cgroup/blkio/docker  && \
chown rootless /sys/fs/cgroup/hugetlb/docker  && \
chown rootless /sys/fs/cgroup/perf_event/docker  && \
chown rootless /sys/fs/cgroup/freezer/docker
"