Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Does Cloudera Directory have roadmap to support Spot Instances in AWS?

Does Cloudera Directory have roadmap to support Spot Instances in AWS?

New Contributor

It is a good solution to use AWS spot instances for cost saving purpose.

Could it be possible to add spot instances option in future version?

5 REPLIES 5

Re: Does Cloudera Directory have roadmap to support Spot Instances in AWS?

Contributor
Hi,

Thanks for your feedback! Spot instance support is on the Cloudera Director
roadmap. Stay tuned for a future release with this feature and please keep
your feedback coming in the meantime.

If you have other experiences or requests that you'd like to share, we'd be
eager to hear from you.

Re: Does Cloudera Directory have roadmap to support Spot Instances in AWS?

New Contributor

Thanks your quick reply.

We are using spot instancs as our main hadoop workers for now. It will be prefect that Couldera Directory can support this feature.

Could you have any possible time when the feature will be released, and we are very happy to participate in test process.

Re: Does Cloudera Directory have roadmap to support Spot Instances in AWS?

New Contributor

Is this supported yet in Cloudera Director? I'm running this API call:

 

 

http://localhost:7189/api/v4/environments/C5-Reference-AWS%20Environment%20/templates/instances

{
"bootstrapScript": "", 
"config": {
    "image": "ami-414b7271", 
    "instanceNamePrefix": "cloudera-director-i-046dd5d9", 
    "rootVolumeSizeGB": "50", 
    "rootVolumeType": "gp2", 
    "securityGroupsIds": "sg-d69f1fb0", 
    "subnetId": "subnet-6a24dc32", 
    "type": "m3.xlarge", 
    "useSpotInstances": true,
    "spotBidUSDPerHr": ".631"
}, 
"image": "ami-414b7271", 
"name": "worker", 
"normalizeInstance": true, 
"tags": {}, 
"type": "m3.xlarge"
}

 

This results in the following error:

 

 

{
  "validationErrors": [
    "Only certain Linux platforms are supported for use with Spot instances. See the \"Supported Distributions and Resource Requirements\" section of the Cloudera Director User Guide. Invalid owner Id for AMI ami-32aa2702: 309956199498 (RedHat)"
  ],
  "conditions": [
    {
      "type": "ERROR",
      "scope": "aws.com.cloudera.director.aws.ec2.EC2Provider.template",
      "key": "image",
      "message": "Only certain Linux platforms are supported for use with Spot instances. See the \"Supported Distributions and Resource Requirements\" section of the Cloudera Director User Guide. Invalid owner Id for AMI ami-32aa2702: 309956199498 (RedHat)"
    }
  ]
}

 

Is there a way to support spot instances in AWS with Red Hat AMIs?

Re: Does Cloudera Directory have roadmap to support Spot Instances in AWS?

Expert Contributor

Unfortunately, this is an EC2 limitation, not a Cloudera Director limitation:

https://aws.amazon.com/partners/redhat/faqs/

 

EC2 support for Spot instances varies by region, instance type, OS, and even by AMI for the same OS, and there is no API to determine whether a specific combination will work. Cloudera Director includes validation for some combinations that are known not to work, but in general you may have to experiment. I know that as of the last time I tried it, in us-west-1, the c4.4xlarge instance type worked with the semi-official CentOS 6 AMI ami-ac5f2fcc [CentOS 6 - with updates (HVM)]. I suspect that the corresponding CentOS 7.2 AMI would work, although I have heard anecdotally of problems with CentOS 7.1.

 

Be sure to read the Cloudera Director documentation about Spot use carefully.

 

Re: Does Cloudera Directory have roadmap to support Spot Instances in AWS?

New Contributor

Thanks! I did get spot instances to work on the same version of the AMI in us-west-2.