Reply
Explorer
Posts: 11
Registered: ‎04-20-2018

Impala daemon crash - CDH 5.14.4

We had an Impala Daemon crash and the traces from the dump generated are mentioned below.

 

Note :

We are currently using CDH 5.14.4. However we had also seen the same issue when we were using the previous version as well namely 5.14.1. We did check the memory and other system level metrics and they looked fine. We could not figure out much from the log below and such crashes has been happening every now and then for us. It would be of good help if someone throws light into the issue. Please let us know in case more information is needed.


Thread 1 (Thread 0x7f6ef14f6700 (LWP 15999)):
#0 0x00007f7c95316207 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f7c953178f8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f7c97638185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#3 0x00007f7c977da593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#4 0x00007f7c9763d68f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#5 0x00007f7c97633be3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#6 <signal handler called>
No symbol table info available.
#7 0x00007f7bdcdaa830 in ?? ()
No symbol table info available.
#8 0x0000000000fddd0f in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) ()
No symbol table info available.
#9 0x0000000000b74d6d in impala::FragmentInstanceState::Open() ()
No symbol table info available.
#10 0x0000000000b763ab in impala::FragmentInstanceState::Exec() ()
No symbol table info available.
#11 0x0000000000b65b38 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
No symbol table info available.
#12 0x0000000000d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
No symbol table info available.
#13 0x0000000000d173c4 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
No symbol table info available.
#14 0x000000000128fada in thread_proxy ()
No symbol table info available.
#15 0x00007f7c956b4dd5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#16 0x00007f7c953deb3d in clone () from /lib64/libc.so.6
No symbol table info available.

Cloudera Employee
Posts: 355
Registered: ‎07-29-2015

Re: Impala daemon crash - CDH 5.14.4

I took a quick look and it doesn't obviously match a known issue and we don't have too much more to go on. If you know which query it is or have other context please let us know. Also if you have minidump crash reports (.dmp files) from the crash I can take a look to see if it's a known issue. If you have a support contract and raise a ticket we can take a deeper dive into it as well.
Explorer
Posts: 11
Registered: ‎04-20-2018

Re: Impala daemon crash - CDH 5.14.4

Hi @Tim Armstrong,

 

We did try but were not able to zero in on the query to reproduce the crash. In fact, all the queries which ran around the time of crash ran fine when they were re-run.

 

I am attaching the mini dump and the complete stacktrace from the dump if at all they could hint at something.

 

The files can be downloaded here 

https://drive.google.com/drive/folders/1Rms93IVZaTuaSGGXE1yXbJeq64nV8Pbz?usp=sharing

 

Also this crash is not a one off instance and had happened multiple times(with different stack traces though) over the past few weeks.

 

Announcements