One of Impala daemon hangs and crashes after launching some queries.
There is an exception in the stacktrace of core dump :
Thread 1 (Thread 0x7f114a377700 (LWP 4369)):
#0 0x00007f60c8c42625 in raise () from /lib64/libc.so.6
#1 0x00007f60c8c43d8d in abort () from /lib64/libc.so.6
#2 0x00007f60c94fca7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x00007f60c94fabd6 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x00007f60c94fac03 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x00007f60c94fad22 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6 0x000000000086449c in void boost::throw_exception<boost::thread_resource_error>(boost::thread_resource_error const&) ()
#7 0x00000000009941e5 in start_thread (this=<value optimized out>, functor=<value optimized out>) at /usr/local/include/boost/thread/detail/thread.hpp:182
It blocks all other queries.
When the problem happens, the daemon has huge amount of virtual memory :
impalad : Mem VIRT : 345G, Mem RES : 9276M
Impala version : 1.4.1
Cluster : 5 nodes with 6 cores and 64G of RAM (5 impalad)
Does anyone know how to fix this ?
Thank you in advance.
This looks like it's some kind of resource exhaustion error: too many threads or running out of memory unexpectedly - possibly related to the number of concurrent queries or the system configuration. There have been a lot of fixes and improvements since Impala 1.4.1 so it's hard to say if it's related to a known or fixed issue.
Thank you for your answer. Actually we don't have the choice about Impala version because this is the most recent version with mapr distribution we use.