HDP 2.5 Cluster Planning

[#1]

Hi All,

I have two physical servers, each sever have 8 core and 128G RAM.

I plan deploy HDP 2.5 cluster on those server.

• Masters -- HDFS NameNode, YARN ResourceManager, and HBase Master. >> 64G RAM
• Slave1 -- HDFS DataNodes, YARN NodeManagers, and HBase RegionServers, KYLIN (query) >> 64G RAM
• Slave2 -- HDFS DataNodes, YARN NodeManagers, and HBase RegionServers, KYLIN (job) >> 64G RAM

When I build cube on Slave2, the performace very bad(High CPU, RAM). But when I check performance on Master and Slave1, I see it free a lot of RAM and CPU.

So is it ok when I take the resource from Master and Slave1 to Slave2. Because In my cluster, only Slave2 have high workload.

Or can you advice my other better design for me.

Thanks,

[#2]

A two node cluster is never a great idea. co locating masters with with slaves is ok when workload is low for POC or simply dev testing. since you are using kylin, it reads hive table, builds cubes and stores in hbase. That activity is very CPU and IO bound. I would say lower your workload throughput (lower volume of data) or add more nodes.

[#3]

@Sunile Manjee

So I need running 2 masters nodes with 1 slaves node, and plan to add more node in future, that right ?.

If I install ESXi in my physical server and spare to multi small nodes(16G Ram each) . Is the performance better than install on big node( physical 64G RAM) ?

If do that I will have a lot of node running, but the RAM, CPU must be depend on physical servers.

[#4]

with 1 slaves node that is only 1 region server. for hbase that is no good. hbase you need min 3 nodes. now if you limit your volumes you can get away with it. just don't expect great performance. all your workload is going to single RS