question Re: Confused with CDH5 YARN/Mr2 container memory settings in Archives of Support Questions (Read Only)

Confused with CDH5 YARN/Mr2 container memory settings

sergey.sheypak566881637 — Fri, 16 Sep 2022 09:11:43 GMT

Hi, I'm trying to switch to CDH5 🙂

I have several nodes each as 32GB ram. Here is my confusion:

1. How to control tasks concurrency? Really, I don't any pools/schedulers right now. I need one huge pool. I don't need any complicity right now, since there is no reason for it. How can I force run all my MR jobs in one huge pool?

2. How to contol memory allocation? I've used mapred.child.java.opts in MR1, in MR2 it doesn't work.

3. I see 100500 setting related to *.memory.mb *.memory.max. I've read description and don't see how they fluence on my MR jobs.

4. Resource manager shows 9 node managers (NM). Each NM has 8Vcores (it's ok, there is 8 HT cores on each node) and 8GB RAM

Why 8GB RAM? I have 32GB per node? How can I change it?

Re: Confused with CDH5 YARN/Mr2 container memory settings

Roy — Tue, 11 Nov 2014 15:05:25 GMT

this link shoud clear your confusion

https://www.linkedin.com/pulse/article/20140706112523-176301000-yarn-resource-allocation

Re: Confused with CDH5 YARN/Mr2 container memory settings

Harsh J — Sun, 30 Nov 2014 12:40:05 GMT

In addition to Roy's link, we do have a guide in our documentation that targets both Developers and Operators/Administrators in terms of what needs to be changed/mapped between MR1 and MR2. You can find these at http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_mapreduce_to_yarn_migrate.html

Re: Confused with CDH5 YARN/Mr2 container memory settings

sergey.sheypak566881637 — Sun, 15 Feb 2015 13:54:05 GMT

Hi, sorry, no luck. Still suffering from MR2/YARN. I have no idea how it works.
Right now I'm getting deadlock several times a day. I have single user which submits jpb. It has huge pool (32*8 mem and 4*CPU) and It has limit for 8 applications at once. Suddenly everything stops.
What does it mean? Who can I get the idea of what's went wrong?