Skip to main content
Ant5566
Bronze III
Mar 1, 2021 6:49 AM UTC
Solved

hive server restarted failed with error stopping schq

The server of hive on tez shut down after it restarted. 

 

We can find error message of hive's log below:

2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer2
2021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:887) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:434) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Web UI has stopped
2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,807 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at acdu02datvm0024.acsdmzuat.cc.cmbchina.cn/172.22.5.22
************************************************************/

 

But I cannot find error message from ZK’s log.

Could anyone please give some advice?

JuliaC
JuliaCAnswer
Gold ⭐️
Sep 1, 2021 5:14 PM UTC

Dear sir,

Which one is correct?

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 

7 replies

GangWar
Platinum ⭐️⭐️
Platinum ⭐️⭐️
Mar 2, 2021 2:53 PM UTC

@Ant5566 This seems a known issue. You need to update tez jars location path in hive.sh under TEZ_JARS property.

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

Restart the service.  

Translate
Cheers! Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
JuliaC
JuliaCAnswer
Gold ⭐️
Sep 1, 2021 5:14 PM UTC

Dear sir,

Which one is correct?

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 

Translate
johndee
Silver ⭐️
Silver ⭐️
Oct 20, 2021 3:17 PM UTC

This one is correct: 

 

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-*:/opt/cloudera/parcels/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

This is a shell script and cannot resolve any jars without the "*". 

Translate
zzeng
Gold ⭐️⭐️
Gold ⭐️⭐️
Jul 9, 2021 6:57 AM UTC

I met similar errors, but with different reasons.

My reason was, I created a leaf queue behind default queue.

Then after Hive restart, it will submit a query to check the health status, however as default queue is not a leaf queue any more, then it went error.

 

My solution was , I removed the child queue behind default queue, then the error resolved.

Translate
RangaReddy
Gold ⭐️
Gold ⭐️
Nov 1, 2021 5:24 AM UTC

Hi 

By default /opt/cloudera/cm-agent/service/hive/hive.sh file, TEZ_JARS property will be

TEZ_JARS="$PARCELS_ROOT/CDH/jars/tez-*:$PARCELS_ROOT/CDH/lib/tez/*.jar:$CONF_DIR/tez-conf"

 We need to update TEZ_JARS property like below:

TEZ_JARS="/opt/cloudera/parcels/CDH/jars/tez-:/opt/cloudera/parcels/CDH/lib/tez/.jar:$CONF_DIR/tez-conf"

After that we need to Restart the service.  

Translate
vi1
Silver ⭐️⭐️
Feb 4, 2023 5:10 AM UTC

After following above steps I'm still not able to start hiveserver2 

Translate