hive server restarted failed with error stopping schq

[#1]

The server of hive on tez shut down after it restarted. 

We can find error message of hive's log below:

2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer2
2021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:887) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:434) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Web UI has stopped
2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,807 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at acdu02datvm0024.acsdmzuat.cc.cmbchina.cn/172.22.5.22
************************************************************/

But I cannot find error message from ZK’s log.

Could anyone please give some advice?

[#2 · Accepted solution]

@Ant5566 This seems a known issue. You need to update tez jars location path in hive.sh under TEZ_JARS property.

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

Restart the service.  

[#3 · Accepted solution]

@Ant5566 This seems a known issue. You need to update tez jars location path in hive.sh under TEZ_JARS property.

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

Restart the service.  

[#4 · Accepted solution]

Dear sir,

Which one is correct?

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

[#5]

This one is correct: 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

This is a shell script and cannot resolve any jars without the "*". 

[#6]

@GangWar this didnt solve the issue for me.

Please provide an alternate solution

[#7]

I met similar errors, but with different reasons.

My reason was, I created a leaf queue behind default queue.

Then after Hive restart, it will submit a query to check the health status, however as default queue is not a leaf queue any more, then it went error.

My solution was , I removed the child queue behind default queue, then the error resolved.

[#8]

Hi 

By default /opt/cloudera/cm-agent/service/hive/hive.sh file, TEZ_JARS property will be

TEZ_JARS="$PARCELS_ROOT/CDH/jars/tez-*:$PARCELS_ROOT/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 We need to update TEZ_JARS property like below:

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

After that we need to Restart the service.  

[#9]

After following above steps I'm still not able to start hiveserver2