Kerberos principal should have 3 parts in sandbox HDP 2.5

[#1]

I enabled kerbosr in sandbox HDP 2.5, but fails to start hive metastore.

This is the princpal setting:

hive.server2.authentication.kerberos.principal hive/_HOST@EXAMPLE.COM

hive.metastore.kerberos.principal hive/_HOST@EXAMPLE.COM

here's the log I see

2017-02-16 11:05:56,541 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6326)) - org.apache.thrift.transport.TTransportException: Kerberos principal should have 3 parts: hive
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.createTransportFactory(HadoopThriftAuthBridge.java:351)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6244)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
2017-02-16 11:05:56,542 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(6159)) - Metastore Thrift Server threw an exception...
org.apache.thrift.transport.TTransportException: Kerberos principal should have 3 parts: hive
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.createTransportFactory(HadoopThriftAuthBridge.java:351)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6244)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

[#2]

@jzhang From the error its suspected that you have not properly defined principal name in configs -

Can you please revisit the configs again

"Kerberos principal should have 3 parts: hive"

It should be - "hive/_HOST@REALM"

Check properties below in HIVE configs -

hive.server2.authentication.kerberos.principal hive.metastore.kerberos.principal

[#3]

Here's the setting:

hive.server2.authentication.kerberos.principal hive/_HOST@EXAMPLE.COM

hive.metastore.kerberos.principal hive/_HOST@EXAMPLE.COM

[#4]

@jzhang could you check the value of hive.metastore.kerberos.principal in hive-site.xml. It should be something like - hive/_HOST@<YOUR REALM>.

[#5]

Here's the setting:

hive.server2.authentication.kerberos.principal hive/_HOST@EXAMPLE.COM

hive.metastore.kerberos.principal hive/_HOST@EXAMPLE.COM

[#6]

@jzhang The _HOST in the principal name gets replaced by the fully qualified domain name of the host at runtime. However, this needs the reverse DNS to be configured correctly on the box. Could you check that?

[#7]

I think it is correct, here's the file /etc/hosts

127.0.0.1       localhost       sandbox.hortonworks.com 
::1     localhost ip6-localhost ip6-loopback 
fe00::0 ip6-localnet 
ff00::0 ip6-mcastprefix 
ff02::1 ip6-allnodes 
ff02::2 ip6-allrouters

[#8]

Any update on this ? This issue is a pretty bad experience.

[#9]

What's the output of "hostname" and "hostname -f" commands on your host where the error occurs? Also, cd to /etc/security/keytabs and run: "klist -kt hive.service.keytab", I suspect the principal stored in the keytab is wrong.

[#10]

Output of klist

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

Output of hostname -f

sandbox.hortonworks.com

[#11]

After reading the hive source code, the correct principle should be hive/sandbox.hortonworks.com@EXAMPLE.COM which has 3 parts, but from the above message, it is hive. And I checked the hive-site.xml, the property hive.metastore.kerberos.principal is correct

[#12]

Anyone else can help ? Thanks

[#13]

@jzhang I was facing similar problem. I added the following to core-site.xml and problem was resolved. Property to watch out for is "hadoop.security.auth_to_local"

<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>

<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

<property>
  <name>hadoop.security.auth_to_local</name>
  <value>
  RULE:[2:$1@$0](hive/.*@.*EXAMPLE.COM)s/.*/hive/
  DEFAULT
  </value>
</property>

[#14]

Hi, I have the exact same problem. I am using the Spark thrift server with the following configuration in hive-site.xml:

<configuration>
<!--
    <property>
       <name>hive.server2.transport.mode</name>
       <value>http</value>
    </property>
-->
        <property>
             <name>hive.server2.authentication</name>
             <value>KERBEROS</value>
        </property>
        <property>
              <name>hive.metastore.kerberos.principal</name>
              <value>thrift/iman@EXAMPLE.COM</value>
        </property>
        <property>
              <name>hive.server2.authentication.kerberos.principal</name>
              <value>thrift/iman@EXAMPLE.COM</value>
        </property>
        <property>
             <name>hive.server2.authentication.kerberos.keytab</name>
             <value>/opt/nginx/iman.keytab</value>
             <description>Keytab file for Spark Thrift server principal</description>  
        </property>
</configuration>

When I start the thrift server by running start-thriftserver.sh, the following error occurs:

18/02/19 18:16:57 ERROR ThriftCLIService: Error starting HiveServer2: could not start ThriftBinaryCLIService
javax.security.auth.login.LoginException: Kerberos principal should have 3 parts: spark
        at org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(HiveAuthFactory.java:148)
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:58)
        at java.lang.Thread.run(Thread.java:748)
18/02/19 18:16:57 INFO HiveServer2: Shutting down HiveServer2

It seems like thrift is mistakingly taking the current user name (spark) as principal name, but if I omit the hive.server2.authentication.kerberos.principal in the config file it would result in "no principal specified" error so it's not missing the configuration entry.

I've had a frustrating time with Kerberos and Apache Thrift. can anyone please help? thanks in advance.