Reply
New Contributor
Posts: 5
Registered: ‎08-11-2014

Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

Hi,

 

I have installed the Cloudera Community Edition 5.14.x using Installation Path B. I have enabled the below mentioned roles on various host.

 

Roles:

 

Screen Shot 2018-05-08 at 12.13.02.png

 

Problem:

Yarn process is eating all the CPU available on the node.

 

Screen Shot 2018-05-08 at 11.10.26.png

 

 

 

 

 

 

 

 

 

 

The process if running as /tmp/java -c /tmp/w.conf

 

I can't find the /tmp/java binary.

 

Screen Shot 2018-05-08 at 12.20.29.png

 

 

 

 

 

 

 

 

 

 

Below are the configuration values changed to fix the problem, but it didn't work.

 

{"defaultFairSharePreemptionThreshold":null,"defaultFairSharePreemptionTimeout":null,"defaultMinSharePreemptionTimeout":null,"defaultQueueSchedulingPolicy":"fair","queueMaxAMShareDefault":null,"queueMaxAppsDefault":null,"queuePlacementRules":[{"create":null,"name":"default","queue":null,"rules":null}],"queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"root","queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"default","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":50.0,"memory":null,"memoryPercent":50.0,"vcores":null},"maxRunningApps":null,"minResources":{"cpuPercent":null,"memory":2048,"memoryPercent":null,"vcores":2},"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"users","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":50.0,"memory":null,"memoryPercent":50.0,"vcores":null},"maxRunningApps":null,"minResources":{"cpuPercent":null,"memory":2048,"memoryPercent":null,"vcores":2},"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":"parent"}],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":50.0,"memory":null,"memoryPercent":50.0,"vcores":null},"maxRunningApps":null,"minResources":{"cpuPercent":null,"memory":2048,"memoryPercent":null,"vcores":2},"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null}],"userMaxAppsDefault":null,"users":[]}

 

Please let me know, how can I prevent yarn eating up all the CPU.

 

Thanks so much,

Surya

 

New Contributor
Posts: 3
Registered: ‎05-08-2018

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

[ Edited ]

I used the following method.

I do not think it is a perfect solution.

 

If you have solved it in #1, you do not need to use #2.

 

- #1.It prevented the access from the outside to the 8088 port.

- #2. I have crontab to kill that process every minute.

  kill $(ps aux | grep 'java -c '| awk '{print $2}')

New Contributor
Posts: 5
Registered: ‎08-11-2014

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

[ Edited ]

Hi Devgate,

 

Thanks for your reply.

 

You have mentioned that,

- #1.It prevented the access from the outside to the 8088 port.

I didn't understand this part. Is JAVA is protecting the yarn itself so that it doesn't available on port 8088?

 

- #2. I have crontab to kill that process every minute.

  kill $(ps aux | grep 'java -c '| awk '{print $2}')

 

I want to know, why its happening and how to prevent that; but ofcourse by not killing the process directly. Are you also getting the same problem?

 

Thanks,

Surya

 

Champion
Posts: 710
Registered: ‎05-16-2016

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

You have way too many roles on the first host . 

I am not sure if that is the host that is exhausting from the cpu if so then try re-deploying some of the roles to other host . 

New Contributor
Posts: 5
Registered: ‎08-11-2014

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

Thanks for your reply.

The Host which is having too many roles, is not having any problem with
that. That I have created as edge node and having 30 GB of memory. The
problem is with the CPU in the rest of hosts where I have assigned Yarn
roles.

What is the actual problem?
New Contributor
Posts: 3
Registered: ‎05-08-2018

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

 

 

I think "/ tmp / java" is not really related to java.
I use the 8088 port as "http: // namenode: 8088 / cluster". The symptom has disappeared since blocking anyopen on this port.

 

 

I had the same symptoms.
I did not know the exact cause, but I shared the results of the above method.

ps: I'm still looking for the root cause.

 

 

 

New Contributor
Posts: 5
Registered: ‎08-11-2014

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

So is right now killing the process if the only solution?
I have allowed all internal ports access to each server, for DMZ  I have allowed the ports provided by the Cloudera Documentation.

@Devgatewrote:

I think "/ tmp / java" is not really related to java.
I use the 8088 port as "http: // namenode: 8088 / cluster". The symptom has disappeared since blocking anyopen on this port.

 

I had the same symptoms.
I did not know the exact cause, but I shared the results of the above method.

ps: I'm still looking for the root cause.

 

 

 


 

New Contributor
Posts: 3
Registered: ‎05-08-2018

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

If you kill that process, it will be regenerated soon.

Putting it all together, I was convinced that this happened via the 8088 port.

 

So I used the above two methods. (The second program was run every minute in the crontab.)

Highlighted
Champion
Posts: 710
Registered: ‎05-16-2016

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

[ Edited ]

You Welcome . 

 

Are you runining Aspire on the rest of the nodes . 

Whats that A stands for ? 

 

I see HS2 hbase LB RM JHS , HBASE  and management services 

 Correct me if i am wrong . 

 

 

Screen Shot 2018-05-10 at 6.34.46 PM.png

New Contributor
Posts: 5
Registered: ‎08-11-2014

Re: Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

A stands for Flume Agent.

This is a development environment, I haven't created the production yet.

Screen Shot 2018-05-13 at 11.02.58.png








I'm running only above roles on data nodes and all 4 cores are occupied. I have not started the data ingestion
or any queries.

 

Screen Shot 2018-05-13 at 11.06.38.png


 

 

 

 

 

 

 

 


@csguna wrote:

You Welcome . 

 

Are you runining Aspire on the rest of the nodes . 

Whats that A stands for ? 

 

I see HS2 hbase LB RM JHS , HBASE  and management services 

 Correct me if i am wrong . 

 

 

Screen Shot 2018-05-10 at 6.34.46 PM.png

 


 

Announcements