• Right now, we have only one manager (node1)

• If we lose it, we lose quorum - and that's very bad!

• Containers running on other nodes will be fine ...

• But we won't be able to get or set anything related to the cluster

• If the manager is permanently gone, we will have to do a manual repair!

• Nobody wants to do that ... so let's make our cluster highly available

With Play-With-Docker:

TOKEN=$(docker swarm join-token -q manager) for N in$(seq 3 5); do
export DOCKER_HOST=tcp://node$N:2375 docker swarm join --token$TOKEN node1:2377
done
unset DOCKER_HOST


## Building our full cluster

• Let's get the token, and use a one-liner for the remaining node with SSH
• Obtain the manager token:

TOKEN=$(docker swarm join-token -q manager)  • Add the remaining node:  ssh node3 docker swarm join --token$TOKEN node1:2377


That was easy.

## Controlling the Swarm from other nodes

• Try the following command on a few different nodes:
docker node ls


On manager nodes:
you will see the list of nodes, with a * denoting the node you're talking to.

On non-manager nodes:
you will get an error message telling you that the node is not a manager.

As we saw earlier, you can only control the Swarm through a manager node.

## Play-With-Docker node status icon

• If you're using Play-With-Docker, you get node status icons

• Node status icons are displayed left of the node name

• No icon = no Swarm mode detected
• Solid blue icon = Swarm manager detected
• Blue outline icon = Swarm worker detected

## Dynamically changing the role of a node

• We can change the role of a node on the fly:

docker node promote nodeX → make nodeX a manager
docker node demote nodeX → make nodeX a worker

• See the current list of nodes:

docker node ls

• Promote any worker node to be a manager:

docker node promote <node_name_or_id>


## How many managers do we need?

• 2N+1 nodes can (and will) tolerate N failures
(you can have an even number of managers, but there is no point)
• 1 manager = no failure

• 3 managers = 1 failure

• 5 managers = 2 failures (or 1 failure during 1 maintenance)

• 7 managers and more = now you might be overdoing it for most designs

see Docker's admin guide on node failure and datacenter redundancy

## Why not have all nodes be managers?

• With Raft, writes have to go to (and be acknowledged by) all nodes

• Thus, it's harder to reach consensus in larger groups

• Only one manager is Leader (writable), so more managers ≠ more capacity

• Managers should be < 10ms latency from each other

• These design parameters lead us to recommended designs

## What would McGyver do?

• Keep managers in one region (multi-zone/datacenter/rack)

• Groups of 3 or 5 nodes: all are managers. Beyond 5, seperate out managers and workers

• Groups of 10-100 nodes: pick 5 "stable" nodes to be managers

• Groups of more than 100 nodes: watch your managers' CPU and RAM

• 16GB memory or more, 4 CPU's or more, SSD's for Raft I/O
• otherwise, break down your nodes in multiple smaller clusters

Cloud pro-tip: use separate auto-scaling groups for managers and workers

See docker's "Running Docker at scale" document

## What's the upper limit?

• We don't know!

• Internal testing at Docker Inc.: 1000-10000 nodes is fine

• deployed to a single cloud region

• one of the main take-aways was "you're gonna need a bigger manager"

• Testing by the community: 4700 heterogeneous nodes all over the 'net

• it just works, assuming they have the resources

• more nodes require manager CPU and networking; more containers require RAM

• scheduling of large jobs (70,000 containers) is slow, though (getting better!)

## Real-life deployment methods

Running commands manually over SSH

(lol jk)