Run Elasticsearch Cluster on Spot Instances

Running Elasticsearch cluster is not cheap, as it generally requires a couple of high spec servers to provides both performances and reliabilities.

Below is a typical 3 nodes Elasticsearch cluster that is running on AWS:

• 3 nodes, one in each availability zone.
• each node is both master and data node, and it is formed by ec2 and ebs.
• ec2 provides computing and Elasticsearch binary.
• ebs provides the storage for data (shards).

Lets say we use r3.xlarge (4 CPU/ 13 ECU, 30.5G RAM) for the ec2 instances. Ignoring the storage and data transfer cost, the computing cost of running this cluster is $1.197 per hour (3 node * r3.xlarge hourly cost $0.399 ), that is $28.728 per day, $861.84 per month. Imagine how much it will cost if you run more nodes or use higher spec instance types.

This typical cluster has some degree of fault tolerance on the cluster level – the cluster still functions if one node dies (e.g instance gets terminated accidentally). But it lacks of the node level high availability. It means unless you fix the faulty node or build a new one then put it back to the cluster, the cluster runs at a degraded performance.

As an automation lover, I think we can run the Elasticsearch cluster in a cheaper and more reliable way. Here is the solution I have worked out:

• 3 nodes, one in each availability zone.
• each node is both master and data node, and it is formed by ec2 and ebs.
• ec2 provides computing and uses spot instance as the preferred candidate.
• ebs provides the storage for data (shards) and Elasticsearch binary.
• If spot fleet pool is greater than 1, then shutdown the current on-demand instance and use the cheapest spot instance for the node.
• If spot fleet pool is less than 1 (which means the spot instances are running out of capacity), then sends an alert to sns to trigger the lambda function to fire up a on-demand instance.
• ec2 is associate with the iam role that allows it to attach/detach the ebs volume, and shutdown other ec2 in the same cluster.

Comparing to the original setup, this solution provides the node level high availability. Additionally, it runs at a way much lower cost even with higher spec!

Here is a sample spot prices list for the instances with 4CPU, 30G RAM and above. In my test, the spot fleet request were always fulfilled with r4.xlarge (4CPU/ 13.5 ECU, 30.5G RAM). A better computing power, but much much lower price. So the computing cost of running the cluster is $0.1038 per hour (3 node * r4.xlarge spot hourly cost $0.0346), that is $2.4912 per day,  $74.736 per month. It saves more than 90% of the original cost ($861.84 per month)!!!