Yarn consumes all the CPU (Running /tmp/java -c /tmp/w.conf) Cloudera Version 5.14.x

[#1]

Hi,

I have installed the Cloudera Community Edition 5.14.x using Installation Path B. I have enabled the below mentioned roles on various host.

Roles:

Problem:

Yarn process is eating all the CPU available on the node.

The process if running as /tmp/java -c /tmp/w.conf

I can't find the /tmp/java binary.

Below are the configuration values changed to fix the problem, but it didn't work.

{"defaultFairSharePreemptionThreshold":null,"defaultFairSharePreemptionTimeout":null,"defaultMinSharePreemptionTimeout":null,"defaultQueueSchedulingPolicy":"fair","queueMaxAMShareDefault":null,"queueMaxAppsDefault":null,"queuePlacementRules":[{"create":null,"name":"default","queue":null,"rules":null}],"queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"root","queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"default","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":50.0,"memory":null,"memoryPercent":50.0,"vcores":null},"maxRunningApps":null,"minResources":{"cpuPercent":null,"memory":2048,"memoryPercent":null,"vcores":2},"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"users","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":50.0,"memory":null,"memoryPercent":50.0,"vcores":null},"maxRunningApps":null,"minResources":{"cpuPercent":null,"memory":2048,"memoryPercent":null,"vcores":2},"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":"parent"}],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":50.0,"memory":null,"memoryPercent":50.0,"vcores":null},"maxRunningApps":null,"minResources":{"cpuPercent":null,"memory":2048,"memoryPercent":null,"vcores":2},"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null}],"userMaxAppsDefault":null,"users":[]}

Please let me know, how can I prevent yarn eating up all the CPU.

Thanks so much,

Surya

[#2]

I used the following method.

I do not think it is a perfect solution.

If you have solved it in #1, you do not need to use #2.

- #1.It prevented the access from the outside to the 8088 port.

- #2. I have crontab to kill that process every minute.

  kill $(ps aux | grep 'java -c '| awk '{print $2}')

[#3]

Hi Devgate,

Thanks for your reply.

You have mentioned that,

- #1.It prevented the access from the outside to the 8088 port.

I didn't understand this part. Is JAVA is protecting the yarn itself so that it doesn't available on port 8088?

- #2. I have crontab to kill that process every minute.

  kill $(ps aux | grep 'java -c '| awk '{print $2}')

I want to know, why its happening and how to prevent that; but ofcourse by not killing the process directly. Are you also getting the same problem?

Thanks,

Surya

[#4]

I think "/ tmp / java" is not really related to java.
I use the 8088 port as "http: // namenode: 8088 / cluster". The symptom has disappeared since blocking anyopen on this port.

I had the same symptoms.
I did not know the exact cause, but I shared the results of the above method.

ps: I'm still looking for the root cause.

[#5]

So is right now killing the process if the only solution?

I have allowed all internal ports access to each server, for DMZ  I have allowed the ports provided by the Cloudera Documentation.

---

@Devgatewrote:

I think "/ tmp / java" is not really related to java.
I use the 8088 port as "http: // namenode: 8088 / cluster". The symptom has disappeared since blocking anyopen on this port.

I had the same symptoms.
I did not know the exact cause, but I shared the results of the above method.

ps: I'm still looking for the root cause.

---

[#6]

If you kill that process, it will be regenerated soon.

Putting it all together, I was convinced that this happened via the 8088 port.

So I used the above two methods. (The second program was run every minute in the crontab.)

[#7]

You have way too many roles on the first host . 

I am not sure if that is the host that is exhausting from the cpu if so then try re-deploying some of the roles to other host . 

[#8]

Thanks for your reply.

The Host which is having too many roles, is not having any problem with
that. That I have created as edge node and having 30 GB of memory. The
problem is with the CPU in the rest of hosts where I have assigned Yarn
roles.

What is the actual problem?

[#9]

You Welcome . 

Are you runining Aspire on the rest of the nodes . 

Whats that A stands for ? 

I see HS2 hbase LB RM JHS , HBASE  and management services 

 Correct me if i am wrong . 

[#10]

A stands for Flume Agent.

This is a development environment, I haven't created the production yet.










I'm running only above roles on data nodes and all 4 cores are occupied. I have not started the data ingestion
or any queries.

---

@csguna wrote:

You Welcome . 

Are you runining Aspire on the rest of the nodes . 

Whats that A stands for ? 

I see HS2 hbase LB RM JHS , HBASE  and management services 

 Correct me if i am wrong . 

---

[#11]

Hi,

I am facing same issue in Cloudera 5.13.0 version. Its a 3 Nodes cluster and on all the Nodes the CPU is going to 100%. I also put the script in crontab as mentioned, It doent work and still the cpu is going high.

Any other indicators ?

Regards

Vivek

[#12]

Thi may be off topic, but I found the similar article.

https://labitacoranet.wordpress.com/2018/05/16/forensic-analysis-of-a-cryptocurrency-mining-attack-i...

[#13]

Hi Tatsuo,

Thanks for the Link.. Very Useful Article.. for the Companies/People who are working in Cloud...

Regards

Vivek

[#14]

---

@VivekM wrote:

Hi,

I am facing same issue in Cloudera 5.13.0 version. Its a 3 Nodes cluster and on all the Nodes the CPU is going to 100%. I also put the script in crontab as mentioned, It doent work and still the cpu is going high.

Any other indicators ?

Regards

Vivek

---

I also dind't found any accecpted solution yet. Mean time I'm using cron to kill the process. I have put below entry in the crontab and the CPU is fine due to that.

* * * * * sudo kill $(ps aux | grep 'java -c '| awk '{print $2}')

[#15]

Hi Surya,

This is workaround is useful...Thanks.

Regards

Vivek

[#16]

Hi,

I solved a similar problem with these commands: 

sudo -u yarn crontab -r; sudo kill $(ps aux | grep 'java -c '| awk '{print $2}'); sudo rm -rf /tmp/java; sudo rm -rf /tmp/ppl3

Need to clear cron of yarn user, kill the "java" process and remove that file