Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hive server restarted failed with error stopping schq

avatar
New Contributor

The server of hive on tez shut down after it restarted. 

 

We can find error message of hive's log below:

2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer2
2021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:887) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:434) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Web UI has stopped
2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,807 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at acdu02datvm0024.acsdmzuat.cc.cmbchina.cn/172.22.5.22
************************************************************/

 

But I cannot find error message from ZK’s log.

Could anyone please give some advice?

2 ACCEPTED SOLUTIONS

avatar
Master Guru

@Ant5566 This seems a known issue. You need to update tez jars location path in hive.sh under TEZ_JARS property.

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

Restart the service.  


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

avatar
Explorer

Dear sir,

Which one is correct?

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 

View solution in original post

7 REPLIES 7

avatar
Master Guru

@Ant5566 This seems a known issue. You need to update tez jars location path in hive.sh under TEZ_JARS property.

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

Restart the service.  


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

Dear sir,

Which one is correct?

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 

avatar
Cloudera Employee

This one is correct: 

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

This is a shell script and cannot resolve any jars without the "*". 

avatar
New Contributor

@GangWar this didnt solve the issue for me.

Please provide an alternate solution

avatar
Expert Contributor

I met similar errors, but with different reasons.

My reason was, I created a leaf queue behind default queue.

Then after Hive restart, it will submit a query to check the health status, however as default queue is not a leaf queue any more, then it went error.

 

My solution was , I removed the child queue behind default queue, then the error resolved.

avatar
Super Collaborator

Hi 

By default /opt/cloudera/cm-agent/service/hive/hive.sh file, TEZ_JARS property will be

TEZ_JARS="$PARCELS_ROOT/CDH/jars/tez-*:$PARCELS_ROOT/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 We need to update TEZ_JARS property like below:

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

After that we need to Restart the service.  

avatar
Explorer

After following above steps I'm still not able to start hiveserver2