Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

[#1]

Hi ,

I am using a Kerberized Cluster in which Hive Metastore and Hive Server starts but stops after few minutes. When I check the logs of hive I get following error messages:

2017-09-20 06:06:00,514 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(5946)) - Metastore Thrift Server threw an exception... org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083. at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:109) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:91) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:83) at org.apache.hadoop.hive.metastore.TServerSocketKeepAlive.<init>(TServerSocketKeepAlive.java:34) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6001) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5942) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 2017-09-20 06:06:00,520 INFO [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:run(5931)) - Shutting down hive metastore. 2017-09-20 06:07:00,165 ERROR [pool-4-thread-200]: server.TThreadPoolServer (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client? at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:75) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) 2017-09-20 06:07:01,631 ERROR [pool-4-thread-200]: server.TThreadPoolServer (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client? at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:75) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)

Please help.

[#2]

@Pooja Kamle

Is there something already running on port 9083? What is the output of "netstat -nlp | grep 9083" ?

[#3]

Hi @mqureshi

Nothing is running on port 9083.

netstat -nlp | grep 9083 gives no output.

[#4]

@Pooja Kamle

Check if your metastore is running or not? See your mysql process. That might be down.

[#5]

Hi, @mqureshi

My mysql process is up. The issue was missing keytabs. Now, my HIVE service is up and running. Thanks for your reply.

[#6 · Accepted solution]

@Pooja Kamle

Do you have some error messages in hiveserver2.log?

[#7]

Hi @Geoffrey Shelton Okot

Thanks for your reply. I saw hiverserver2.log. According to the error messages in the logs the keytabs were missing.

After creating keytabs, HIVE service is started successfully.