Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cloud Sizing for AWS, Azure and Google - What are best instance types to use?

avatar
Master Guru

What are the best instance types for HDP nodes (master, data, edge)?

I found a number of instance types that may work.

Looking at

TCO

https://awstcocalculator.com/

Rough Pricing

https://calculator.s3.amazonaws.com/index.html

Amazon Types

http://www.ec2instances.info/

Amazon EC2 Types

https://aws.amazon.com/ec2/instance-types/

1 ACCEPTED SOLUTION

avatar
Master Guru

@Timothy Spann "best" is a subjective term when it comes to public cloud providers. I would take your typically on perm profile and back port it to the AWS EC2 profiles. From a pricing perspective you can further spread the master/datanodes services to small boxes based on cost difference. For example ZK uses low ram. Then you can find a small 2-4gig box for your ZK quorum. Take the typical on perm requirements and back port into 1:M VM on your cloud provider.

For example

Master Nodes - Multipule of m4.4xlarge, or r3.4xlarge

Data nodes - i2.4xlarge or d2.4xlarge

Storm nodes - c4.4xlarge

Spark - x1.32xlarge

GPU processing - g2.2xlarge

View solution in original post

2 REPLIES 2

avatar
Master Guru

@Timothy Spann "best" is a subjective term when it comes to public cloud providers. I would take your typically on perm profile and back port it to the AWS EC2 profiles. From a pricing perspective you can further spread the master/datanodes services to small boxes based on cost difference. For example ZK uses low ram. Then you can find a small 2-4gig box for your ZK quorum. Take the typical on perm requirements and back port into 1:M VM on your cloud provider.

For example

Master Nodes - Multipule of m4.4xlarge, or r3.4xlarge

Data nodes - i2.4xlarge or d2.4xlarge

Storm nodes - c4.4xlarge

Spark - x1.32xlarge

GPU processing - g2.2xlarge

avatar
Master Guru

Thanks for the analysis. Does anyone have similiar sizings for Google and Azure: