How to prevent CPU high load

[#1]

In my cluster, when I running an Yarn app, my CPU of each node get very high.

I want to prevent this issue. I configure yarn.nodemanager.resource.percentage-physical-cpu-limit=80(%)

But nodes still run out of CPU ( some time is 200% CPU cause I can not ssh to this) and also cause high CPU, hbase region server force to stop.

I need limit CPU usage of nodes that mean when I run Yarn app, the CPU can not run out of resource. How can I do that.

Thanks

[#2]

How big of a yarn app?

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_hive-performance-tuning/bk_hive-performan...

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-installation/content/determ...

[#3]

@Timothy Spann

Yarn app not big so much, I run Apache Kylin and buid cubes. When running, Yarn app follow my memory setting. But for CPU, even when I set limit for CPU, it always exceed over my limit. I mean when I set CPU limit 80%, but when app running , it reach to 100% even 200%

[#4]

You will need to enable CPU scheduling. by default yarn only accounts for ram. if you enable cpu scheduling, each yarn app will ram for and cpu, and yarn will provide containers based on what is available per your ram and CPU limits. if cpu scheduling is not enabled, very difficult to limit.

[#5]

@Sunile Manjee

Do you have any documents relate to this (cpu scheduling) ?

[#6]

Here is documenation on enabling cpu scheduling

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_yarn_resource_mgt/content/ch_cpu_schedul...

[#7]

Hi @Hoang Le

Use the following doc to configure cpu scheduling:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_yarn-resource-management/content/ch_cpu_s...

But it's also recommended to configure cgroups for this to be effective:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_yarn-resource-management/content/enabling...

However, these can be a bit of pain.

Most of the time, increasing the container size (map, reduce and tez) would reduce the CPU load.

Setting YARN container memory values is a balancing act between memory (RAM), processors (CPU cores), and disks so that processing is not constrained by any one of these cluster resources.

However, if the application usage pattern is known, the containers can be tuned to influence and maximize the resources utilization. For example, if CPU usage and load average are too large, an increase in the container size reduces the number of containers allocated per node. A good rule of thumb is not to have the load average bigger than 4 times the number of physical core.

Then a likely cause for having large amounts of CPU used is Garbage Collections and especially continuous Garbage Collection which would be helped by a larger container.

Increase by 50% the following and observe any changes:

tez.task.resource.memory.mb

hive.tez.container.size

mapreduce.map.memory.mb

mapreduce.reduce.memory.mb

[#8]

Please look at the Java heap size. Try increasing heap size first. This will give you more memory to prevent many GC cycles. You want high CPU usage for your application just not for GC and other applications.

Also, please check to see if other applications are running. Running Linux top command may help to see if other tasks are running and where the CPU is being used.

[#9]

@Hoang Le, did you ever get this resolved? I am interesting in to know how to fix this issue too?