Skip to main content
bkandalkar
bkandalkar
Gold ⭐️
January 25, 2019
Solved

Hive metadata server failing

Hello Team,

After enabling kerberos on HDP-2.6, Hive metadata server is failing. Getting following error in hive metastore log:

2019-01-25 03:58:28,880 ERROR [pool-7-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -128
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1849)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: Invalid status -128
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more

How to solve this? Please suggest.

Thanks,

Bhushan

bkandalkar
bkandalkarAnswer
Gold ⭐️
January 28, 2019

Resolved issue by installing hive client on Hive metastore machine.

View in thread

3 replies

bkandalkar
bkandalkarAuthor
Gold ⭐️
January 27, 2019

@Sindhu @Geoffrey Shelton Okot @Sandeep Nemuri ... please suggest

Translate
Shelton
Shelton
Gold ⭐️⭐️⭐️
January 27, 2019

@Bhushan Kandalkar

Can you start with the following checks to investigate the SaslTransport issues, first the hive keytab

# ll /etc/security/keytabs/hive.service.keytab

Desired output see ownership and permission bits !!!

-r--r----- 1 hive hadoop 353 Oct 11 10:49 /etc/security/keytabs/hive.service.keytab

Check the Hive--->Configs--->Advanced hive-site check the hive.server2.authentication.kerberos.principal

Desired output

hive/_HOST@REALM

This should match the entry in the Kerberos database, validate by running on the KDC server the below command as root user

# kadmin.local
kadmin.local: listprincs

Desired output hive/$FQDN@REALM

Lastly, Can you regenerate the keytabs using the Ambari Kerberos wizard the restart the cluster

HTH

Translate
bkandalkar
bkandalkarAuthorAnswer
Gold ⭐️
January 28, 2019

Resolved issue by installing hive client on Hive metastore machine.

Translate
Terms & ConditionsAccessibility statement