Upgrading to overlay2 on Ubuntu 20.04 arm64 - get "invalid argument, driver not supported"

[bburns]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Docker should let me upgrade from devicemapper to overlay2 storage engine.

Actual behavior

Get error when starting Docker -

failed to mount overlay: invalid argument storage-driver=overlay2
driver not supported

It's my understanding that Docker uses overlay2 by default, but I seem to be stuck with devicemapper. I was having problems with that eating up disk space, so wanted to switch to overlay2.

Steps to reproduce the behavior

$ sudo systemctl stop docker
$ sudo rm -rf /var/lib/docker

$ sudo nano /etc/docker/daemon.json
$ sudo cat /etc/docker/daemon.json
{
  "storage-driver": "overlay2"
}

$ sudo systemctl start docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

$ sudo dockerd --debug
INFO[2022-05-25T06:48:33.025968569Z] Starting up
DEBU[2022-05-25T06:48:33.027335891Z] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2022-05-25T06:48:33.028909628Z] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
DEBU[2022-05-25T06:48:33.031405107Z] Golang's threads limit set to 27360
INFO[2022-05-25T06:48:33.034216750Z] parsed scheme: "unix"                         module=grpc
INFO[2022-05-25T06:48:33.034358207Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-05-25T06:48:33.034593580Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-05-25T06:48:33.035834028Z] ClientConn switching balancer to "pick_first"  module=grpc
DEBU[2022-05-25T06:48:33.034378623Z] metrics API listening on /var/run/docker/metrics.sock
INFO[2022-05-25T06:48:33.046671395Z] parsed scheme: "unix"                         module=grpc
INFO[2022-05-25T06:48:33.046786019Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-05-25T06:48:33.046875852Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-05-25T06:48:33.046921352Z] ClientConn switching balancer to "pick_first"  module=grpc
DEBU[2022-05-25T06:48:33.049799494Z] Using default logging driver json-file
DEBU[2022-05-25T06:48:33.049948826Z] [graphdriver] trying provided driver: overlay2
DEBU[2022-05-25T06:48:33.049809202Z] processing event stream                       module=libcontainerd namespace=plugins.moby
ERRO[2022-05-25T06:48:33.054543121Z] failed to mount overlay: invalid argument     storage-driver=overlay2
DEBU[2022-05-25T06:48:33.055641820Z] Cleaning up old mountid : start.
failed to start daemon: error initializing graphdriver: driver not supported

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:44:28 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:05 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.12
 Storage Driver: devicemapper
  Pool Name: docker-0:28-398286-pool
  Pool Blocksize: 65.54kB
  Base Device Size: 10.74GB
  Backing Filesystem: ext4
  Udev Sync Supported: true
  Data file: /dev/loop0
  Metadata file: /dev/loop1
  Data loop file: /var/lib/docker/devicemapper/devicemapper/data
  Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
  Data Space Used: 240.1MB
  Data Space Total: 107.4GB
  Data Space Available: 23.87GB
  Metadata Space Used: 17.47MB
  Metadata Space Total: 2.147GB
  Metadata Space Available: 2.13GB
  Thin Pool Minimum Free Space: 10.74GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.167 (2019-11-30)
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.60
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 3.745GiB
 Name: l99-682719a73ffd
 ID: ZHEX:YJ4R:I4A4:TV4G:U6HO:5FCI:CXUA:QGQ5:BK57:LNMF:ZNB3:234Y
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

Additional environment details (AWS, VirtualBox, physical, etc.)

$ uname -a
Linux l99-682719a73ffd 5.10.60 #1 SMP PREEMPT Wed Oct 27 14:12:50 CST 2021 aarch64 aarch64 aarch64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

[MRIIOT]

start fresh

systemctl stop docker
cd /var/lib/docker
rm -rf *
mkdir -p /var/lib/docker/devicemapper/devicemapper
dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=250
#https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/
systemctl start docker

[tukusejssirs]

The error might happen when the root file system (/) is itself an overlay. You can check it using df -h / and looking for the file system column value.

Based on this, [o]verlay on top of overlay is not supported by the kernel, which would be the case when the root FS is an overlay and you want to run Docker in an overlay on top of it.

One can work this around using additional partition (formatted to, e.g., ext4) mounted at /var/lib/docker (or /var/lib/docker/overlay2). Don’t forget to stop Docker service, copy the data from /var/lib/docker somewhere, mount the partition, copy the data back and update /etc/fstab to make sure the partition is mounted at system startup.

[volkertb]

Did you by any chance enable the casefold feature in the ext4 partition that holds /var/lib/docker? This is a relatively new feature that allows specific directories to be made case-insensitive through a file/directory attribute. I'm not sure how many distros already enable the casefold feature in ext4 partitions by default in new installs.

I just ran into a very similar issue after I enabled casefold on my ext4 partitions. The cause is apparently a bug in overlayfs. See https://bugzilla.kernel.org/show_bug.cgi?id=216471

Some workarounds are suggested in this answer on StackExchange.

Sharing this here also for reference to others who find this GitHub issue while searching for the same error(s).