Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Node Manager frequently down

Node Manager frequently down

Expert Contributor

Hi Folks,

 

We are using HDP 2.6.5 version with HBase, oozie. We are getting "java.lang.OutOfMemoryError: unable to create new native thread" error in our logs:

 

 

 

2019-09-22 05:56:37,297 INFO container.ContainerImpl - Container container_e56_1565687270063_197657_01_001489 transitioned from LOCALIZING to LOCALIZED
2019-09-22 05:56:37,299 FATAL event.AsyncDispatcher - Error in dispatcher thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:118)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:55)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,300 INFO ipc.Server - Auth successful for appattempt_1565687270063_197657_000001 (auth:SIMPLE)
2019-09-22 05:56:37,303 FATAL yarn.YarnUncaughtExceptionHandler - Thread Thread[AsyncDispatcher event handler,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException

 

I have gone through multiple links, But i didn't get valuable output.

Ouput of max thread: cat /proc/sys/kernel/threads-max
902309

ulimit output: ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 451154
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 451154
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

 

Could some help me to resolve the issue?

 

Regards,

Vinay K

 

4 REPLIES 4

Re: Node Manager frequently down

Mentor

@vinayk 

Error in dispatcher thread java.lang.OutOfMemoryError: unable to create new native thread. This just means that the OS has reached the limits of the number of threads. Please can you run your current Output of max thread:
# cat /proc/sys/fs/file-max

Check the global
# ulimit

You could set it to unlimited by

# ulimit -c unlimited

​Maximum Open Files Requirements

The recommended maximum number of open file descriptors is 10000, or more. To check the current value set for the maximum number of open file descriptors, execute the following shell commands on each host:

# ulimit -Sn

#ulimit -Hn

If the output is not greater than 10000, run the following command to set it to a suitable default:

# ulimit -n 10000

 

Now retry  restarting your hbase  and revert 

Re: Node Manager frequently down

Expert Contributor

Hi @Shelton 

 

Thanks for give your precious time.

As you have mentioned, already value is set:

#cat /proc/sys/fs/file-max
11534502

#ulimit
unlimited

#ulimit -Sn
128000

#ulimit -Hn
128000

Each node in Hbase Cluster are already running with these settings.

 

Noticed: vm.swappiness value is 60, also sometime swap uses is more than 80%.

#free -m

Swap: 7167 6066 1101

 

Looking for other solution.

 

Regards,

Vinay K

Re: Node Manager frequently down

Mentor

@vinayk 

Can you reset the swappiness.  In order to prevent the system from swapping processes too frequently, but still allow for emergency swapping , the recommendation is now to set swappiness to 1 on Linux systems you value of 60 is the default 

please do the below changes and revert

echo "1" > /proc/sys/vm/swappiness

To ensure the swappiness is set appropriately on reboot, use the following command:

echo "vm.swappiness=1" >> /etc/sysctl.conf

.  Happy hadooping

Highlighted

Re: Node Manager frequently down

Mentor

@vinayk 

Any updates?