Migrating: Zookeeper

Rational

Zookeeper is a distributed key-value store. In this article we will go through the online migration procedure. Migrating a distributed system can be very easy on paper and very hard on practice but doable never the less.

My method for migrating as with most of the distributed systems, is by expanding and then shrinking.

Expand

The first step is to deploy 3 nodes with same Zookeeper vesrion installed.

For this example we are going to have 3 new nodes with details:

IDs: 13, 14 ,15 
IPs: 10.0.0.13, 10.0.0.14, 10.0.0.15

and an existing cluster with details:

IDs: 10, 11 ,12 
IPs: 10.0.0.10, 10.0.0.11, 10.0.0.12

Configure the new zookeeper nodes with old nodes connection details:

dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=200
tickTime=2000
# This cluster
server.13=10.0.0.13:2888:3888
server.14=10.0.0.14:2888:3888
server.15=10.0.0.15:2888:3888
# OLD Cluster
server.10=10.0.0.10:2888:3888
server.11=10.0.0.11:2888:3888
server.12=10.0.0.12:2888:3888
initLimit=20
syncLimit=10

Restart the new nodes one by one (Rolling fashion) in order to connect to the existing cluster. They will not be followers yet.

Configure olds nodes with new nodes connections details:

dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=200
tickTime=2000
# This cluster
server.13=10.0.0.13:2888:3888
server.14=10.0.0.14:2888:3888
server.15=10.0.0.15:2888:3888
# OLD Cluster
server.10=10.0.0.10:2888:3888
server.11=10.0.0.11:2888:3888
server.12=10.0.0.12:2888:3888
initLimit=20
syncLimit=10

Roll restart the old nodes so that new nodes join the cluster as followers now. Restart followers first and the leader afterwards. To check if a node is either leader or followers use the following command:

root@prod-zk-0:~# echo status | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
 /10.11.31.142:51394[1](queued=0,recved=322992,sent=325374)
Latency min/avg/max: 0/0/607
Received: 1096044
Sent: 1110742
Connections: 8
Outstanding: 0
Zxid: 0x4a0003f341
Mode: leader
Node count: 4250
Proposal sizes last/min/max: 36/32/12090

Check that all three new nodes are now folowers.

Let’s shrink

Now that we have hopefully:) a 6 node zk cluster with 5 followers and 1 leader lets just remove the old nodes.

Stop the 2 old followers and restart the leader so that a rebalance is performed and a new leader is elected from the new nodes. Reconfigure the new nodes to remove the old cluster ips:

dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=200
tickTime=2000
# This cluster
server.13=10.0.0.13:2888:3888
server.14=10.0.0.14:2888:3888
server.15=10.0.0.15:2888:3888
initLimit=20
syncLimit=10

Now roll restart each node from the new cluster. Finally stop the remaining old node.

Fin

Thats all! Drink a beer and enjoy your new Zookeeper cluster!