CDH Hardware requirements for a small cluster

[#1]

Hello!

Please help me about minimal hardware requirements for out small cluster.

We decided to make very small production cluster with high availability for archiving purposes based on Cloudera CDH 6.3.3 (community version)

Storage size planning as about of 10-20 TiB

Workload planning:

- every 2 minutes ETL from external oracle to local parquet about of 500-1000 rows of data

- periodically (very rarely) analytic queries to hive about search through all of parquets

- periodically (very rarely) run spark ad-hoc tasks with goals same as above

Components:

- Cloudera manager

- HDFS

- Hive

- Hue

- Impala

- Spark

- Yarn with MR2

- ZooKeeper

- Streamsets parcel (as a part of cloudera)

We want to use only 3 hosts (not more) and disaster of any of this host must not crash all system.

So we plan to place all of above components to all of the hosts.

In another words, each component will be on each host.

Is it normal and available or someone may advise another alignment?

We also want to know if we can place HDFS namenode and cloudera manager on only 2 hosts or this components also better to put to all three hosts?

And, finally, which minimal requirements of RAM, CPU and disk storage to each of this three hosts?

Big thanks in advance!

[#2]

Ma be we will exclude impala from this list for resources economy. Hive is enough for a while.

[#3 · Accepted solution]

@tol2000 This guide may be a good start for you. 

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_hardware_requirements.h...

[#4]

Thank you. I have started thinking about it exactly from this resource 🙂 Not all is clear for me for a while, but now I have +- ability of imaging about this question, it's all very individual, I understand. Thank you. This question is closed )