Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hive server restarted failed with error stopping schq

avatar
New Contributor

The server of hive on tez shut down after it restarted. 

 

We can find error message of hive's log below:

2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer2
2021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:887) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:434) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Web UI has stopped
2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,807 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at acdu02datvm0024.acsdmzuat.cc.cmbchina.cn/172.22.5.22
************************************************************/

 

But I cannot find error message from ZK’s log.

Could anyone please give some advice?

2 ACCEPTED SOLUTIONS

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
7 REPLIES 7

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Cloudera Employee

This one is correct: 

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

This is a shell script and cannot resolve any jars without the "*". 

avatar
New Contributor

@GangWar this didnt solve the issue for me.

Please provide an alternate solution

avatar
Expert Contributor

I met similar errors, but with different reasons.

My reason was, I created a leaf queue behind default queue.

Then after Hive restart, it will submit a query to check the health status, however as default queue is not a leaf queue any more, then it went error.

 

My solution was , I removed the child queue behind default queue, then the error resolved.

avatar
Master Collaborator

Hi 

By default /opt/cloudera/cm-agent/service/hive/hive.sh file, TEZ_JARS property will be

TEZ_JARS="$PARCELS_ROOT/CDH/jars/tez-*:$PARCELS_ROOT/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 We need to update TEZ_JARS property like below:

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

After that we need to Restart the service.  

avatar
Explorer

After following above steps I'm still not able to start hiveserver2