Hi Folks,
We are using HDP 2.6.5 version with HBase, oozie. We are getting "java.lang.OutOfMemoryError: unable to create new native thread" error in our logs:
2019-09-22 05:56:37,297 INFO container.ContainerImpl - Container container_e56_1565687270063_197657_01_001489 transitioned from LOCALIZING to LOCALIZED
2019-09-22 05:56:37,299 FATAL event.AsyncDispatcher - Error in dispatcher thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:118)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncher.java:55)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,300 INFO ipc.Server - Auth successful for appattempt_1565687270063_197657_000001 (auth:SIMPLE)
2019-09-22 05:56:37,303 FATAL yarn.YarnUncaughtExceptionHandler - Thread Thread[AsyncDispatcher event handler,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)
2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException2019-09-22 05:56:37,308 INFO util.ExitUtil - Halt with status -1 Message: HaltException
I have gone through multiple links, But i didn't get valuable output.
Ouput of max thread: cat /proc/sys/kernel/threads-max 902309
ulimit output: ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 451154 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 128000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 451154 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Could some help me to resolve the issue?
Regards,
Vinay K
... View more
