Support Questions

Find answers, ask questions, and share your expertise

Yarn configuration best practices- Zeppelin and Hive hang

Rising Star

Hi,

I have HDP2.6 on 2 master nodes and 3 client nodes.

Zeppelin and Hive keep hang. I assume it should be related to the resource manager settings in Yarn.

Please advise me the best practices for Yarn configuration.

1 REPLY 1

Super Collaborator

Hi @Sanaz Janbakhsh,

Thats clearly look like a sign of issue with resource management.

if you can see the sessions being held in Accepted state despite being in running in state, then ,

you can tweak the configuration by adjusting the User Limit Factor & Maximum Applications yarn queue config, so that it will leverage number of concurrent applications you can run.

Please refer the following documentation which give details description on setting and adjusting Scheduler Configurations.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-views/content/setting_up_queues.h...

From Documentation to be considered for changing the settings.

  • User Limit Factor -- The default value of "1" means that any single user in the queue can at maximum only occupy the queue’s configured capacity. This prevents users in a single queue from monopolizing resources across all queues in a cluster. Setting the value to "2" would restrict the queue's users to twice the queue’s configured capacity. Setting it to a value of 0.5 would restrict any user from using resources beyond half of the queue capacity.
  • Minimum User Limit -- This property can be used to set the minimum percentage of resources allocated to each queue user. For example, to enable equal sharing of the queue capacity among five users, you would set this property to 20%.
  • Maximum Applications -- To avoid system-thrash due to an unmanageable load -- caused either by malicious users, or accidentally -- the Capacity Scheduler enables you to place a static, configurable limit on the total number of concurrently active (both running and pending) applications at any one time.
  • Maximum AM Resource -- The limit for running applications in any specific queue is a fraction of this total limit, proportional to its capacity. This is a hard limit, which means that once this limit is reached for a queue, any new applications submitted to that queue will be rejected, and clients will have to wait and retry later.

On the side node please that you will not over allocate concurrent applications so that nothing will progress from scheduler.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.