Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive metadata server failing

avatar
Contributor

Hello Team,

After enabling kerberos on HDP-2.6, Hive metadata server is failing. Getting following error in hive metastore log:

2019-01-25 03:58:28,880 ERROR [pool-7-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -128
       at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
       at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609)
       at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:360)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1849)
       at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606)
       at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: Invalid status -128
       at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
       at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184)
       at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
       at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
       at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
       at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
       ... 10 more

How to solve this? Please suggest.

Thanks,

Bhushan

1 ACCEPTED SOLUTION

avatar
Contributor

Resolved issue by installing hive client on Hive metastore machine.

View solution in original post

3 REPLIES 3

avatar
Contributor

avatar
Master Mentor

@Bhushan Kandalkar

Can you start with the following checks to investigate the SaslTransport issues, first the hive keytab

# ll /etc/security/keytabs/hive.service.keytab 

Desired output see ownership and permission bits !!!

-r--r----- 1 hive hadoop 353 Oct 11 10:49 /etc/security/keytabs/hive.service.keytab 

Check the Hive--->Configs--->Advanced hive-site check the hive.server2.authentication.kerberos.principal

Desired output

hive/_HOST@REALM 

This should match the entry in the Kerberos database, validate by running on the KDC server the below command as root user

# kadmin.local 
kadmin.local: listprincs 

Desired output hive/$FQDN@REALM

Lastly, Can you regenerate the keytabs using the Ambari Kerberos wizard the restart the cluster

HTH

avatar
Contributor

Resolved issue by installing hive client on Hive metastore machine.