Getting inside a container

Objectives

On a traditional server or VM, we sometimes need to:

• analyze the disks (by removing them or rebooting with a rescue system).

In this chapter, we will see how to do that with containers.

Getting a shell

Every once in a while, we want to log into a machine.

In an perfect world, this shouldn't be necessary.

• You need to install or update packages (and their configuration)?

Use configuration management. (e.g. Ansible, Chef, Puppet, Salt...)

• You need to view logs and metrics?

Collect and access them through a centralized platform.

In the real world, though ... we often need shell access!

Not getting a shell

Even without a perfect deployment system, we can do many operations without getting a shell.

• Installing packages can (and should) be done in the container image.

• Configuration can be done at the image level, or when the container starts.

• Dynamic configuration can be stored in a volume (shared with another container).

• Logs written to stdout are automatically collected by the Docker Engine.

• Other logs can be written to a shared volume.

• Process information and metrics are visible from the host.

Let's save logging, volumes ... for later, but let's have a look at process information!

Viewing container processes from the host

If you run Docker on Linux, container processes are visible on the host.

$ps faux | less  • Scroll around the output of this command. • You should see the jpetazzo/clock container. • A containerized process is just like any other process on the host. • We can use tools like lsof, strace, gdb ... To analyze them. What's the difference between a container process and a host process? • Each process (containerized or not) belongs to namespaces and cgroups. • The namespaces and cgroups determine what a process can "see" and "do". • Analogy: each process (containerized or not) runs with a specific UID (user ID). • UID=0 is root, and has elevated privileges. Other UIDs are normal users. We will give more details about namespaces and cgroups later. Getting a shell in a running container • Sometimes, we need to get a shell anyway. • We could run some SSH server in the container ... • But it is easier to use docker exec. $ docker exec -ti ticktock sh

• This creates a new process (running sh) inside the container.

• This can also be done "manually" with the tool nsenter.

Caveats

• The tool that you want to run needs to exist in the container.

• Some tools (like ip netns exec) let you attach to one namespace at a time.

(This lets you e.g. setup network interfaces, even if you don't have ifconfig or ip in the container.)

• Most importantly: the container needs to be running.

• What if the container is stopped or crashed?

Getting a shell in a stopped container

• A stopped container is only storage (like a disk drive).

• We cannot SSH into a disk drive or USB stick!

• We need to connect the disk to a running machine.

• How does that translate into the container world?

Analyzing a stopped container

As an exercise, we are going to try to find out what's wrong with jpetazzo/crashtest.

docker run jpetazzo/crashtest


The container starts, but then stops immediately, without any output.

What would MacGyver™ do?

First, let's check the status of that container.

docker ps -l


Viewing filesystem changes

• We can use docker diff to see files that were added / changed / removed.
docker diff <container_id>

• The container ID was shown by docker ps -l.

• We can also see it with docker ps -lq.

• The output of docker diff shows some interesting log files!

Accessing files

• We can extract files with docker cp.
docker cp <container_id>:/var/log/nginx/error.log .

• Then we can look at that log file.
cat error.log


(The directory /run/nginx doesn't exist.)

Exploring a crashed container

• We can restart a container with docker start ...

• ... But it will probably crash again immediately!

• We cannot specify a different program to run with docker start

• But we can create a new image from the crashed container

docker commit <container_id> debugimage

• Then we can run a new container from that image, with a custom entrypoint
docker run -ti --entrypoint sh debugimage


Obtaining a complete dump

• We can also dump the entire filesystem of a container.

• This is done with docker export.

• It generates a tar archive.

docker export <container_id> | tar tv


This will give a detailed listing of the content of the container.