Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impalad hangs / crashes : boost thread ressource error

Impalad hangs / crashes : boost thread ressource error

New Contributor

Hi everyone,

 

One of Impala daemon hangs and crashes after launching some queries.

 

There is an exception in the stacktrace of core dump :
Thread 1 (Thread 0x7f114a377700 (LWP 4369)):
#0 0x00007f60c8c42625 in raise () from /lib64/libc.so.6
#1 0x00007f60c8c43d8d in abort () from /lib64/libc.so.6
#2 0x00007f60c94fca7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x00007f60c94fabd6 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x00007f60c94fac03 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x00007f60c94fad22 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6 0x000000000086449c in void boost::throw_exception<boost::thread_resource_error>(boost::thread_resource_error const&) ()
#7 0x00000000009941e5 in start_thread (this=<value optimized out>, functor=<value optimized out>) at /usr/local/include/boost/thread/detail/thread.hpp:182

 

It blocks all other queries.
When the problem happens, the daemon has huge amount of virtual memory :
impalad : Mem VIRT : 345G, Mem RES : 9276M


Impala version : 1.4.1
Cluster : 5 nodes with 6 cores and 64G of RAM (5 impalad)

Does anyone know how to fix this ?

 

Thank you in advance.

 

Best regards,

 

2 REPLIES 2

Re: Impalad hangs / crashes : boost thread ressource error

Master Collaborator

This looks like it's some kind of resource exhaustion error: too many threads or running out of memory unexpectedly - possibly related to the number of concurrent queries or the system configuration. There have been a lot of fixes and improvements since Impala 1.4.1 so it's hard to say if it's related to a known or fixed issue.

Re: Impalad hangs / crashes : boost thread ressource error

New Contributor

Thank you for your answer. Actually we don't have the choice about Impala version because this is the most recent version with mapr distribution we use.