Node Manager frequently down

[#1]

Hi Folks,

We are using HDP 2.6.5 version with HBase, oozie. We are getting "java.lang.OutOfMemoryError: unable to create new native thread" error in our logs:

2019-09-22 05:56:37,297 INFO container.ContainerImpl - Container container_e56_1565687270063_197657_01_001489 transitioned from LOCALIZING to LOCALIZED
2019-09-22 05:56:37,299 FATAL event.AsyncDispatcher - Error in dispatcher thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:118)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:55)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,300 INFO ipc.Server - Auth successful for appattempt_1565687270063_197657_000001 (auth:SIMPLE)
2019-09-22 05:56:37,303 FATAL yarn.YarnUncaughtExceptionHandler - Thread Thread[AsyncDispatcher event handler,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException

I have gone through multiple links, But i didn't get valuable output.

Ouput of max thread: cat /proc/sys/kernel/threads-max
902309

ulimit output: ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 451154
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 451154
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Could some help me to resolve the issue?

Regards,

Vinay K

[#2]

@vinayk 

Error in dispatcher thread java.lang.OutOfMemoryError: unable to create new native thread. This just means that the OS has reached the limits of the number of threads. Please can you run your current Output of max thread:
# cat /proc/sys/fs/file-max

Check the global
# ulimit

You could set it to unlimited by

# ulimit -c unlimited

​Maximum Open Files Requirements

The recommended maximum number of open file descriptors is 10000, or more. To check the current value set for the maximum number of open file descriptors, execute the following shell commands on each host:

# ulimit -Sn

#ulimit -Hn

If the output is not greater than 10000, run the following command to set it to a suitable default:

# ulimit -n 10000

Now retry  restarting your hbase  and revert 

[#3]

Hi @Shelton 

Thanks for give your precious time.

As you have mentioned, already value is set:

#cat /proc/sys/fs/file-max
11534502

#ulimit
unlimited

#ulimit -Sn
128000

#ulimit -Hn
128000

Each node in Hbase Cluster are already running with these settings.

Noticed: vm.swappiness value is 60, also sometime swap uses is more than 80%.

#free -m

Swap: 7167 6066 1101

Looking for other solution.

Regards,

Vinay K

[#4]

@vinayk 

Can you reset the swappiness.  In order to prevent the system from swapping processes too frequently, but still allow for emergency swapping , the recommendation is now to set swappiness to 1 on Linux systems you value of 60 is the default 

please do the below changes and revert

echo "1" > /proc/sys/vm/swappiness

To ensure the swappiness is set appropriately on reboot, use the following command:

echo "vm.swappiness=1" >> /etc/sysctl.conf

.  Happy hadooping

[#5]

@vinayk 

Any updates?