Impala daemon stuck at Ready Check

[#1]

Hello,

We have a 3 node cluster, on one our node we see Impala Daemon in concerning state with error as (Impala Daemon Ready Check - The readiness of the Impala Daemon to process queries is not known.), while the other daemon works fine. Restarting this daemon fixes the issue momentarily, but the issue soon comes back. Attached is part of the error log.

E0710 15:36:17.214027 28015 impala-server.cc:1454] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
I0710 15:36:18.856057 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33567
I0710 15:36:20.856436 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33568
I0710 15:36:22.856806 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33569
I0710 15:36:24.857154 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33570
I0710 15:36:25.126052 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 140.06 MB
I0710 15:36:25.791110 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 110.30 MB
I0710 15:36:26.857511 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33571
I0710 15:36:28.857841 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33572
I0710 15:36:30.858582 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33573
I0710 15:36:31.781077 28015 jni-util.cc:211] java.lang.NoClassDefFoundError: org/antlr/runtime/CharStream
	at org.apache.impala.analysis.SlotRef.<init>(SlotRef.java:47)
	at org.apache.impala.analysis.CUP$SqlParser$actions.case588(SqlParser.java:14267)
	at org.apache.impala.analysis.CUP$SqlParser$actions.CUP$SqlParser$do_action(SqlParser.java:3909)
	at org.apache.impala.analysis.SqlParser.do_action(SqlParser.java:1309)
	at java_cup.runtime.lr_parser.parse(lr_parser.java:587)
	at org.apache.impala.catalog.View.init(View.java:150)
	at org.apache.impala.catalog.View.loadFromThrift(View.java:131)
	at org.apache.impala.catalog.Table.fromThrift(Table.java:271)
	at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:398)
	at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:282)
	at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:169)
	at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:227)
	at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:191)
Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.CharStream
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 13 more
I0710 15:36:31.816437 28015 status.cc:125] NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
    @           0x95b1e9  impala::Status::Status()
    @           0xca2552  impala::JniUtil::GetJniExceptionMsg()
    @           0xba2b28  impala::Frontend::UpdateCatalogCache()
    @           0xbc0469  impala::ImpalaServer::CatalogUpdateCallback()
    @           0xc613e3  impala::StatestoreSubscriber::UpdateState()
    @           0xc670d3  impala::StatestoreSubscriberThriftIf::UpdateState()
    @          0x10f2ea8  impala::StatestoreSubscriberProcessor::process_UpdateState()
    @          0x10f20e4  impala::StatestoreSubscriberProcessor::dispatchCall()
    @           0x92b8bc  apache::thrift::TDispatchProcessor::process()
    @           0xafbc4f  apache::thrift::server::TAcceptQueueServer::Task::run()
    @           0xaf6545  impala::ThriftThread::RunRunnable()
    @           0xaf7322  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @           0xd15193  impala::Thread::SuperviseThread()
    @           0xd158d4  boost::detail::thread_data<>::run()
    @          0x12919aa  (unknown)
    @     0x7f44d68a9dd5  start_thread
    @     0x7f44d65d2ead  __clone

Any help / guidance is much appericated 

Regards

Wert

[#2]

Hi Wert ,

From what I can see, Impala does not use class org.antlr.runtime.CharStream directly, but Hive does, I am wondering what are the errors you can find from Catalogd and HiveMetaStore server logs? Just to confirm if the issue is really coming from Hive?

Also, what version of CDH are you using? Is it only affecting 1 out of 3 hosts? How do you install CDH? Through CM or manual process?

Cheers
Eric

[#3]

If I had to guess, the CDH installation is somehow broken and missing jar files. Impala depends on antlr so won't be able to run if that isn't present. The JARs should be part of the CDH parcel, e.g. in /opt/cloudera/parcels/CDH-<version>/lib/impala/lib