05-10-2017 09:19 PM
We have a requirement to migrate data from ODS (plus some social media, web analytics etc) into Hadoop for which we need to create a cluster. Please find below the details:
Node Type Disk in TB's (7200 RPM) RAM Cores
NN + JN + RM +ZK 1(OS) + 2(FSImage & Edit logs) + 1(JN) + 1(ZK) 32 14
StandBy NN + JN Same as NN 32 14
Edge + CM 1 14 4
Cloudera Director node 1 14 4
Data Nodes (4*3TB)
(3 disks of 1 TB per node) 4*3 32 8
(Also one of DN will be JN as well)
1. Can anyone please confirm if I need to change anything ?
2. Is it mandatory to have separate RM node in prod? If yes, what should be its conf?
3. Can I have Director on Edge along with Cloudera Manager?
4. Also, please suggest what should I change(scale down) to set up a Dev env as well ?
05-11-2017 08:06 AM
My initial thought is that is a lot of services running without much RAM on each box.
You mentioned you would be using HBASE and 7.5 TB of data. Are you planning on having all that data stored in Hbase?