I was working on a project called SpotOn last year. The basically idea is to use Spot fleet whenever possible. If the pool size is lower than the threshold, then CloudWatch will trigger a Lambda function to scale up the on-demand EC2 to pick up the workloads. Check this out if you want to know more: Run ElasticSearch cluster on Spot Instance.
The SpotOn solution worked very well, however we may not need it anymore as AWS just recently launched a new service named EC2 Fleet. Currently it is only available through CLI or API.
According to the AWS documentations, it looks really promising.
“An EC2 Fleet contains the configuration information to launch a fleet—or group—of instances. In a single API call, a fleet can launch multiple instance types across multiple Availability Zones, using the On-Demand Instance, Reserved Instance, and Spot Instance purchasing models together. Using EC2 Fleet, you can define separate On-Demand and Spot capacity targets, specify the instance types that work best for your applications, and specify how Amazon EC2 should distribute your fleet capacity within each purchasing model.
The EC2 Fleet attempts to launch the number of instances that are required to meet the target capacity specified in your request. The fleet also attempts to maintain its target Spot capacity if your Spot Instances are interrupted due to a change in Spot prices or available capacity”
The highlights of EC2 fleet is that it allows user to mix the use of different instance type and instance purchasing model (spot, on-demand and reserved) in just a single API call. And by default it selects the most cost effective combination of the instance types and availability zones. Also users is able to modify the submitted request.
Reference:
https://aws.amazon.com/blogs/aws/amazon-ec2-spot-fleet-api-manage-thousands-of-instances-with-one-request/
https://aws.amazon.com/blogs/aws/ec2-fleet-manage-thousands-of-on-demand-and-spot-instances-with-one-request/
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-fleet.html#EC2-fleet-limitations
Jackie Chen's IT Workshop 