Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive metadata server failing

Solved Go to solution

Hive metadata server failing

Contributor

Hello Team,

After enabling kerberos on HDP-2.6, Hive metadata server is failing. Getting following error in hive metastore log:

2019-01-25 03:58:28,880 ERROR [pool-7-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -128
       at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
       at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609)
       at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:360)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1849)
       at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606)
       at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: Invalid status -128
       at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
       at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184)
       at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
       at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
       at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
       at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
       ... 10 more

How to solve this? Please suggest.

Thanks,

Bhushan

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hive metadata server failing

Contributor

Resolved issue by installing hive client on Hive metastore machine.

3 REPLIES 3

Re: Hive metadata server failing

Contributor

Re: Hive metadata server failing

Mentor

@Bhushan Kandalkar

Can you start with the following checks to investigate the SaslTransport issues, first the hive keytab

# ll /etc/security/keytabs/hive.service.keytab 

Desired output see ownership and permission bits !!!

-r--r----- 1 hive hadoop 353 Oct 11 10:49 /etc/security/keytabs/hive.service.keytab 

Check the Hive--->Configs--->Advanced hive-site check the hive.server2.authentication.kerberos.principal

Desired output

hive/_HOST@REALM 

This should match the entry in the Kerberos database, validate by running on the KDC server the below command as root user

# kadmin.local 
kadmin.local: listprincs 

Desired output hive/$FQDN@REALM

Lastly, Can you regenerate the keytabs using the Ambari Kerberos wizard the restart the cluster

HTH

Highlighted

Re: Hive metadata server failing

Contributor

Resolved issue by installing hive client on Hive metastore machine.