Help! An unremovable Docker network!

For a client, I've set up a collection of containers in a Docker environment on a Linux VM.

This VM was a beast: lots of RAM, amazingly fast, but... the disk drive was sized quite niggardly. The end result: the machine ran out of disk space and one of my containers crashed.

Nothing that a docker system prune --all can't fix, I thought. Oh, 20 GB freed by removing unused contains, images, ...? Great, let's create a new container.

Houston, we have a problem...

So, when I execute docker-compose up, Docker complains that the network already exists. What? Weird?

Indeed: docker network ls shows me that the network is there, alive and kicking. No problem, let's remove that stale network then and start with a clean slate.

docker network rm {network} gives me the following error:

Error response from daemon: network scheduledintegrationtestsprofile_uvms id 19d69d29f4751e19fb5a716889ec7ea9c66b1d27cdfd64d98cb4ed8b589d09f5 has active endpoints.

What's the issue?

What? I'm quite sure that no containers are running on this network. Let's inspect the network, to find out which containers are attached.

I can view the details of a network with docker inspect {network}. For example, with this command I can check which containers are attached to the network. Since all containers that were attached to the network are now removed, I don't expect any container to be attached. Hmm... let's see.

[
    {
       "Name": "blatest123profile_uvms",
        ...
        "Containers": {
            ...
            "Name": "blatest123-activemq",
            ...
        },
    }
]

In the "Containers" section, I can see that 2 containers are still attached. But... these containers do no longer exist. I've effectively removed them with, which I can verify with docker ps -a. I have no idea why the network still thinks they are attached.

The solution

Now that I know which "ghost" containers are still attached, I can force the network to disconnect with this phantom container with:

docker network disconnect -f {network} {containerName}

When you run out of disk space, Docker can behave strangely. I've encountered this issue multiple times.

The real solution? Provide enough disk space to your VMs/machines. In the mean time, this workaround hopefully gets you unblocked for what you really want to do: deliver kick-ass software.