YARN&MR Configuration: "HDP Configuration Utils" or "Ambari Stack Advisor"?

[#1]

For configuring/tuning YARN & MR memory settings:

• the HDP Configuration Utility is referenced in the HDP documentation.
• and the Ambari "Stack Advisor" provides similar configuration.

Which is preferred? Do they overlap? Are they maintained similarly (such that they don't conflict)? And so on ...

[#2 · Accepted solution]

@Sean Roberts - the answer to that question really depends on the version of Ambari that you're using. There are multiple tools that can be used to calculate the appropriate memory settings for Hive/Tez/YARN/MR and I would treat them in this order of preference based on two things: 1. the frequency of update, and 2. the amount of context that the tool has for its calculations:

1. SmartSense - The memory-related rules are updated very frequently, at least once a quarter and have the most context as it has all of the SmartSense diagnostic information at its disposal and can take into account actual use versus configured use of services (cores, spindles, memory, other services running on that machine, other 3rd party utilities being run on that machine)
2. Stack Advisor - Updated frequently, but tied to Ambari releases so depends on if the customer is using Ambari and if they are which specific version of Ambari and how up to date it is 1.7 vs 2.0 vs 2.1, etc..
3. HDP Configuration Utility - Most basic and least frequently updated, but if the customer does not have Ambari or SmartSense and is manually deploying HDP is better than nothing.