Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH Hardware requirements for a small cluster

Solved Go to solution
Highlighted

CDH Hardware requirements for a small cluster

New Contributor
Hello!
 
Please help me about minimal hardware requirements for out small cluster.
 
We decided to make very small production cluster with high availability for archiving purposes based on Cloudera CDH 6.3.3 (community version)

Storage size planning as about of 10-20 TiB

Workload planning:

- every 2 minutes ETL from external oracle to local parquet about of 500-1000 rows of data
- periodically (very rarely) analytic queries to hive about search through all of parquets
- periodically (very rarely) run spark ad-hoc tasks with goals same as above

Components:

- Cloudera manager
- HDFS
- Hive
- Hue
- Impala
- Spark
- Yarn with MR2
- ZooKeeper
- Streamsets parcel (as a part of cloudera)

We want to use only 3 hosts (not more) and disaster of any of this host must not crash all system.

So we plan to place all of above components to all of the hosts.
In another words, each component will be on each host.
Is it normal and available or someone may advise another alignment?

We also want to know if we can place HDFS namenode and cloudera manager on only 2 hosts or this components also better to put to all three hosts?

And, finally, which minimal requirements of RAM, CPU and disk storage to each of this three hosts?

Big thanks in advance!
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: CDH Hardware requirements for a small cluster

Master Collaborator

@tol2000 This guide may be a good start for you. 

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_hardware_requirements.h...


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

3 REPLIES 3
Highlighted

Re: CDH Hardware requirements for a small cluster

New Contributor

Ma be we will exclude impala from this list for resources economy. Hive is enough for a while.

Highlighted

Re: CDH Hardware requirements for a small cluster

Master Collaborator

@tol2000 This guide may be a good start for you. 

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_hardware_requirements.h...


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

Highlighted

Re: CDH Hardware requirements for a small cluster

New Contributor

Thank you. I have started thinking about it exactly from this resource Not all is clear for me for a while, but now I have +- ability of imaging about this question, it's all very individual, I understand. Thank you. This question is closed )

Don't have an account?
Coming from Hortonworks? Activate your account here