Reply
New Contributor
Posts: 2
Registered: ‎11-24-2015

Impalad hangs / crashes : boost thread ressource error

Hi everyone,

 

One of Impala daemon hangs and crashes after launching some queries.

 

There is an exception in the stacktrace of core dump :
Thread 1 (Thread 0x7f114a377700 (LWP 4369)):
#0 0x00007f60c8c42625 in raise () from /lib64/libc.so.6
#1 0x00007f60c8c43d8d in abort () from /lib64/libc.so.6
#2 0x00007f60c94fca7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x00007f60c94fabd6 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x00007f60c94fac03 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x00007f60c94fad22 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6 0x000000000086449c in void boost::throw_exception<boost::thread_resource_error>(boost::thread_resource_error const&) ()
#7 0x00000000009941e5 in start_thread (this=<value optimized out>, functor=<value optimized out>) at /usr/local/include/boost/thread/detail/thread.hpp:182

 

It blocks all other queries.
When the problem happens, the daemon has huge amount of virtual memory :
impalad : Mem VIRT : 345G, Mem RES : 9276M


Impala version : 1.4.1
Cluster : 5 nodes with 6 cores and 64G of RAM (5 impalad)

Does anyone know how to fix this ?

 

Thank you in advance.

 

Best regards,

 

Cloudera Employee
Posts: 422
Registered: ‎07-29-2015

Re: Impalad hangs / crashes : boost thread ressource error

This looks like it's some kind of resource exhaustion error: too many threads or running out of memory unexpectedly - possibly related to the number of concurrent queries or the system configuration. There have been a lot of fixes and improvements since Impala 1.4.1 so it's hard to say if it's related to a known or fixed issue.

New Contributor
Posts: 2
Registered: ‎11-24-2015

Re: Impalad hangs / crashes : boost thread ressource error

Thank you for your answer. Actually we don't have the choice about Impala version because this is the most recent version with mapr distribution we use.