Hello,
We have a 3 node cluster, on one our node we see Impala Daemon in concerning state with error as (Impala Daemon Ready Check - The readiness of the Impala Daemon to process queries is not known.), while the other daemon works fine. Restarting this daemon fixes the issue momentarily, but the issue soon comes back. Attached is part of the error log.
E0710 15:36:17.214027 28015 impala-server.cc:1454] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
I0710 15:36:18.856057 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33567
I0710 15:36:20.856436 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33568
I0710 15:36:22.856806 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33569
I0710 15:36:24.857154 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33570
I0710 15:36:25.126052 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 140.06 MB
I0710 15:36:25.791110 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 110.30 MB
I0710 15:36:26.857511 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33571
I0710 15:36:28.857841 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33572
I0710 15:36:30.858582 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33573
I0710 15:36:31.781077 28015 jni-util.cc:211] java.lang.NoClassDefFoundError: org/antlr/runtime/CharStream
at org.apache.impala.analysis.SlotRef.<init>(SlotRef.java:47)
at org.apache.impala.analysis.CUP$SqlParser$actions.case588(SqlParser.java:14267)
at org.apache.impala.analysis.CUP$SqlParser$actions.CUP$SqlParser$do_action(SqlParser.java:3909)
at org.apache.impala.analysis.SqlParser.do_action(SqlParser.java:1309)
at java_cup.runtime.lr_parser.parse(lr_parser.java:587)
at org.apache.impala.catalog.View.init(View.java:150)
at org.apache.impala.catalog.View.loadFromThrift(View.java:131)
at org.apache.impala.catalog.Table.fromThrift(Table.java:271)
at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:398)
at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:282)
at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:169)
at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:227)
at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:191)
Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.CharStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
I0710 15:36:31.816437 28015 status.cc:125] NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
@ 0x95b1e9 impala::Status::Status()
@ 0xca2552 impala::JniUtil::GetJniExceptionMsg()
@ 0xba2b28 impala::Frontend::UpdateCatalogCache()
@ 0xbc0469 impala::ImpalaServer::CatalogUpdateCallback()
@ 0xc613e3 impala::StatestoreSubscriber::UpdateState()
@ 0xc670d3 impala::StatestoreSubscriberThriftIf::UpdateState()
@ 0x10f2ea8 impala::StatestoreSubscriberProcessor::process_UpdateState()
@ 0x10f20e4 impala::StatestoreSubscriberProcessor::dispatchCall()
@ 0x92b8bc apache::thrift::TDispatchProcessor::process()
@ 0xafbc4f apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0xaf6545 impala::ThriftThread::RunRunnable()
@ 0xaf7322 boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0xd15193 impala::Thread::SuperviseThread()
@ 0xd158d4 boost::detail::thread_data<>::run()
@ 0x12919aa (unknown)
@ 0x7f44d68a9dd5 start_thread
@ 0x7f44d65d2ead __clone
Any help / guidance is much appericated
Regards
Wert
