Created on
09-23-2019
06:59 AM
- last edited on
09-23-2019
09:03 AM
by
VidyaSargur
Hi Folks,
We are using HDP 2.6.5 version with HBase, oozie. We are getting "java.lang.OutOfMemoryError: unable to create new native thread" error in our logs:
2019-09-22 05:56:37,297 INFO container.ContainerImpl - Container container_e56_1565687270063_197657_01_001489 transitioned from LOCALIZING to LOCALIZED
2019-09-22 05:56:37,299 FATAL event.AsyncDispatcher - Error in dispatcher thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:118)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:55)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,300 INFO ipc.Server - Auth successful for appattempt_1565687270063_197657_000001 (auth:SIMPLE)
2019-09-22 05:56:37,303 FATAL yarn.YarnUncaughtExceptionHandler - Thread Thread[AsyncDispatcher event handler,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException
I have gone through multiple links, But i didn't get valuable output.
Ouput of max thread: cat /proc/sys/kernel/threads-max
902309
ulimit output: ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 451154
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 451154
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Could some help me to resolve the issue?
Regards,
Vinay K
Created 09-23-2019 11:53 AM
Error in dispatcher thread java.lang.OutOfMemoryError: unable to create new native thread. This just means that the OS has reached the limits of the number of threads. Please can you run your current Output of max thread:
# cat /proc/sys/fs/file-max
Check the global
# ulimit
You could set it to unlimited by
# ulimit -c unlimited
Maximum Open Files Requirements
The recommended maximum number of open file descriptors is 10000, or more. To check the current value set for the maximum number of open file descriptors, execute the following shell commands on each host:
# ulimit -Sn
#ulimit -Hn
If the output is not greater than 10000, run the following command to set it to a suitable default:
# ulimit -n 10000
Now retry restarting your hbase and revert
Created 09-24-2019 06:35 AM
Hi @Shelton
Thanks for give your precious time.
As you have mentioned, already value is set:
#cat /proc/sys/fs/file-max
11534502
#ulimit
unlimited
#ulimit -Sn
128000
#ulimit -Hn
128000
Each node in Hbase Cluster are already running with these settings.
Noticed: vm.swappiness value is 60, also sometime swap uses is more than 80%.
#free -m
Swap: 7167 6066 1101
Looking for other solution.
Regards,
Vinay K
Created 09-26-2019 12:51 PM
Can you reset the swappiness. In order to prevent the system from swapping processes too frequently, but still allow for emergency swapping , the recommendation is now to set swappiness to 1 on Linux systems you value of 60 is the default
please do the below changes and revert
echo "1" > /proc/sys/vm/swappiness
To ensure the swappiness is set appropriately on reboot, use the following command:
echo "vm.swappiness=1" >> /etc/sysctl.conf
. Happy hadooping
Created 09-30-2019 10:20 AM
Any updates?