Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

kamlepooja_raje — Wed, 20 Sep 2017 12:22:24 GMT

Hi ,

I am using a Kerberized Cluster in which Hive Metastore and Hive Server starts but stops after few minutes. When I check the logs of hive I get following error messages:

2017-09-20 06:06:00,514 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(5946)) - Metastore Thrift Server threw an exception... org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083. at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:109) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:91) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:83) at org.apache.hadoop.hive.metastore.TServerSocketKeepAlive.<init>(TServerSocketKeepAlive.java:34) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6001) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5942) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 2017-09-20 06:06:00,520 INFO [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:run(5931)) - Shutting down hive metastore. 2017-09-20 06:07:00,165 ERROR [pool-4-thread-200]: server.TThreadPoolServer (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client? at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:75) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) 2017-09-20 06:07:01,631 ERROR [pool-4-thread-200]: server.TThreadPoolServer (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client? at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:75) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)

Please help.

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

mqureshi — Wed, 20 Sep 2017 13:21:30 GMT

@Pooja Kamle

Is there something already running on port 9083? What is the output of "netstat -nlp | grep 9083" ?

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

kamlepooja_raje — Wed, 20 Sep 2017 14:05:11 GMT

Hi @mqureshi

Nothing is running on port 9083.

netstat -nlp | grep 9083 gives no output.

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

Shelton — Wed, 20 Sep 2017 14:23:25 GMT

@Pooja Kamle

Do you have some error messages in hiveserver2.log?

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

mqureshi — Thu, 21 Sep 2017 04:28:28 GMT

@Pooja Kamle

Check if your metastore is running or not? See your mysql process. That might be down.

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

kamlepooja_raje — Thu, 21 Sep 2017 18:03:59 GMT

Hi @Geoffrey Shelton Okot

Thanks for your reply. I saw hiverserver2.log. According to the error messages in the logs the keytabs were missing.

After creating keytabs, HIVE service is started successfully.

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

kamlepooja_raje — Thu, 21 Sep 2017 18:05:58 GMT

Hi, @mqureshi

My mysql process is up. The issue was missing keytabs. Now, my HIVE service is up and running. Thanks for your reply.

question Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster in Support Questions

Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster

Re: Hive Metastore and Hive Server starts but stops immediately in a Kerberized cluster