Dealing with stateful services

• Building a stateful service experiment
• Accessing our Redis service easily
• Basic Redis commands
• Local volumes vs. global volumes
• Local volumes
• Where is our data?
• Setting the key again
• Updating a service recreates its containers
• Service volumes are ephemeral by default
• The data is gone again
• Assigning a persistent volume to the container
• Checking that data is now persisted correctly
• Recap
• Cleaning up
• Should I run stateful services in containers?

• First of all, you need to make sure that the data files are on a volume

• Volumes are host directories that are mounted to the container's filesystem

• These host directories can be backed by the ordinary, plain host filesystem ...

• ... Or by distributed/networked filesystems

• In the latter scenario, in case of node failure, the data is safe elsewhere ...

• ... And the container can be restarted on another node without data loss

Building a stateful service experiment

• We will use Redis for this example

• We will expose it on port 10000 to access it easily

• Start the Redis service:

  docker service create --name stateful -p 10000:6379 redis
  

• Check that we can connect to it:

  docker run --net host --rm redis redis-cli -p 10000 info server
  

Accessing our Redis service easily

• Typing that whole command is going to be tedious

• Define a shell alias to make our lives easier:

  alias redis='docker run --net host --rm redis redis-cli -p 10000'
  

• Try it:

  redis info server
  

Basic Redis commands

• Check that the foo key doesn't exist:

  redis get foo
  

• Set it to bar:

  redis set foo bar
  

• Check that it exists now:

  redis get foo
  

Local volumes vs. global volumes

• Global volumes exist in a single namespace

• A global volume can be mounted on any node (bar some restrictions specific to the volume driver in use; e.g. using an EBS-backed volume on a GCE/EC2 mixed cluster)

• Attaching a global volume to a container allows to start the container anywhere
  (and retain its data wherever you start it!)

• Global volumes require extra plugins (Flocker, Portworx...)

• Docker doesn't come with a default global volume driver at this point

• Therefore, we will fall back on local volumes

Local volumes

• We will use the default volume driver, local

• As the name implies, the local volume driver manages local volumes

• Since local volumes are (duh!) local, we need to pin our container to a specific host

• We will do that with a constraint

• Add a placement constraint to our service:

  docker service update stateful --constraint-add node.hostname==$HOSTNAME
  

Where is our data?

• If we look for our foo key, it's gone!

• Check the foo key:

  redis get foo
  

• Adding a constraint caused the service to be redeployed:

  docker service ps stateful
  

Note: even if the constraint ends up being a no-op (i.e. not moving the service), the service gets redeployed. This ensures consistent behavior.

Setting the key again

• Since our database was wiped out, let's populate it again

• Set foo again:

  redis set foo bar
  

• Check that it's there:

  redis get foo
  

Updating a service recreates its containers

• Let's try to make a trivial update to the service and see what happens

• Set a memory limit to our Redis service:

  docker service update stateful --limit-memory 100M
  

• Try to get the foo key one more time:

  redis get foo
  

The key is blank again!

Service volumes are ephemeral by default

• Let's highlight what's going on with volumes!

• Check the current list of volumes:

  docker volume ls
  

• Carry a minor update to our Redis service:

  docker service update stateful --limit-memory 200M
  

Again: all changes trigger the creation of a new task, and therefore a replacement of the existing container; even when it is not strictly technically necessary.

The data is gone again

• What happened to our data?

• The list of volumes is slightly different:

  docker volume ls
  

(You should see one extra volume.)

Assigning a persistent volume to the container

• Let's add an explicit volume mount to our service, referencing a named volume

• Update the service with a volume mount:

    docker service update stateful \
           --mount-add type=volume,source=foobarstore,target=/data
  

• Check the new volume list:

  docker volume ls
  

Note: the local volume driver automatically creates volumes.

Checking that data is now persisted correctly

• Store something in the foo key:

  redis set foo barbar
  

• Update the service with yet another trivial change:

  docker service update stateful --limit-memory 300M
  

• Check that foo is still set:

  redis get foo
  

Recap

• The service must commit its state to disk when being shutdown.red[*]

  (Shutdown = being sent a TERM signal)

• The state must be written on files located on a volume

• That volume must be specified to be persistent

• If using a local volume, the service must also be pinned to a specific node

  (And losing that node means losing the data, unless there are other backups)

.red[*]If you customize Redis configuration, make sure you persist data correctly!
It's easy to make that mistake — Trust me!]

Cleaning up

• Remove the stateful service:

  docker service rm stateful
  

• Remove the associated volume:

  docker volume rm foobarstore
  

Note: we could keep the volume around if we wanted.

Should I run stateful services in containers?

Depending whom you ask, they'll tell you:

• certainly not, heathen!

• we've been running a few thousands PostgreSQL instances in containers ...
  for a few years now ... in production ... is that bad?

• what's a container?

Perhaps a better question would be:

"Should I run stateful services?"

• is it critical for my business?
• is it my value-add?
• or should I find somebody else to run them for me?