Memory settings when server have different memory configurations

[#1]

Hi community,

Spending weeks setting up and HDP 2.5 LAB at work (doing it part-time) I finally got it up and running. So my little LAB has 4 servers now :

• 3 x Dell T3500 (6 cores) with 72 GB RAM and two disks (1=OS, 1 for APPS)
• 1 x Clone computer (3 cores) with 16 GB RAM and two disks also.

Since the "clone" computer is a little behind the others I reserved it for Datanode only server. One of the Dell is another Datanode and the other 2 are mixed "management" (YARN, Name, SName, Hive, HBase, MySQL, etc...) and "Datanode".

After installation I realized that not all the memory is being used. 6-7 GB on each server at most. So I started investigating how I could use my memory more efficiently and potentially increase my performances.

Found this article "Determine memory" but it seems to assume that all nodes have the same configuration.

Can I set my memory parameters has per this article and forget that one of the servers is only 16GB ?

[#2 · Accepted solution]

@Marc Mazerolle Ambari has an option to use configuration groups for heterogenous configurations, be it memory or something else. Here's more info on that, you'll be able to set specific and proportional memory properties for whichever hardware configuration there is. https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Ambari_Users_Guide/content/_using_host_c...

[#3]

Makes sense. Once I create two groups then I run the script in article once per group I guess. I'll try this sometime this week and let you know how it worked out. Thanks.