Kerberos principal should have 3 parts in sandbox HDP 2.5

[#1]

I enabled kerbosr in sandbox HDP 2.5, but fails to start hive metastore.

This is the princpal setting:

hive.server2.authentication.kerberos.principal hive/_HOST@EXAMPLE.COM

hive.metastore.kerberos.principal hive/_HOST@EXAMPLE.COM

here's the log I see

2017-02-16 11:05:56,541 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6326)) - org.apache.thrift.transport.TTransportException: Kerberos principal should have 3 parts: hive
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.createTransportFactory(HadoopThriftAuthBridge.java:351)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6244)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
2017-02-16 11:05:56,542 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(6159)) - Metastore Thrift Server threw an exception...
org.apache.thrift.transport.TTransportException: Kerberos principal should have 3 parts: hive
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.createTransportFactory(HadoopThriftAuthBridge.java:351)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6244)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

[#2]

@jzhang From the error its suspected that you have not properly defined principal name in configs -

Can you please revisit the configs again

"Kerberos principal should have 3 parts: hive"

It should be - "hive/_HOST@REALM"

Check properties below in HIVE configs -

hive.server2.authentication.kerberos.principal hive.metastore.kerberos.principal

[#3]

Here's the setting:

hive.server2.authentication.kerberos.principal hive/_HOST@EXAMPLE.COM

hive.metastore.kerberos.principal hive/_HOST@EXAMPLE.COM

[#4]

@jzhang could you check the value of hive.metastore.kerberos.principal in hive-site.xml. It should be something like - hive/_HOST@<YOUR REALM>.

[#5]

Here's the setting:

hive.server2.authentication.kerberos.principal hive/_HOST@EXAMPLE.COM

hive.metastore.kerberos.principal hive/_HOST@EXAMPLE.COM

[#6]

@jzhang The _HOST in the principal name gets replaced by the fully qualified domain name of the host at runtime. However, this needs the reverse DNS to be configured correctly on the box. Could you check that?

[#7]

I think it is correct, here's the file /etc/hosts

127.0.0.1       localhost       sandbox.hortonworks.com 
::1     localhost ip6-localhost ip6-loopback 
fe00::0 ip6-localnet 
ff00::0 ip6-mcastprefix 
ff02::1 ip6-allnodes 
ff02::2 ip6-allrouters

[#8]

Any update on this ? This issue is a pretty bad experience.

[#9]

What's the output of "hostname" and "hostname -f" commands on your host where the error occurs? Also, cd to /etc/security/keytabs and run: "klist -kt hive.service.keytab", I suspect the principal stored in the keytab is wrong.

[#10]

Output of klist

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

4 03/07/17 03:25:16 hive/sandbox.hortonworks.com@EXAMPLE.COM

Output of hostname -f

sandbox.hortonworks.com