Can not invoke hdfs command after invoking webhdfs operation in secure cluster

[#1]

Here's what I did in my secure cluster:

1. Invokding webhdfs command: "curl -s -i --negotiate -u:anyUser http://sandbox.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS" (this works)

2. Invoking hdfs command: "hadoop fs -ls /"

But get the following error. (I need to kdestroy and kinti again to renew the ticket to make it work). This is weird, how can I make the webhdfs command not affect the hdfs command ?

16/02/23 03:38:41 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "sandbox.hortonworks.com/10.0.2.15"; destination host is: "sandbox.hortonworks.com":8020;

The only difference after step 1 in klist is that I have 2 extra HTTP principal, but I still have my principal jeff@EXAMPLE.COM

Default principal: jeff@EXAMPLE.COM Valid starting Expires Service principal 02/23/16 03:20:05 02/24/16 03:20:05 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 02/23/16 03:20:05 02/23/16 03:20:10 02/24/16 03:20:05 HTTP/sandbox.hortonworks.com@ renew until 02/23/16 03:20:05 02/23/16 03:20:10 02/24/16 03:20:05 HTTP/sandbox.hortonworks.com@EXAMPLE.COM renew until 02/23/16 03:20:05

[#2]

@jzhang

This is related to not having correct ticket from KDC. You can test by getting a ticket from hdfs keytab. kinit with hdfs headless and see if hdfs -ls works or not

[#3]

Yeah, I see this should due to ticket issue. But I don't understand why the webhdfs command corrupt my ticket. If I only invoke step 2, everything is OK.

[#4]

@jzhang You will be running curl from the client node and hdfs from edge or master node.

[#5]

No, I am running this in the same node. Actually I did it in sandbox. Only one node.

[#6]

@jzhang That's exactly my point that REST access will from the client so you should not worry about the kerberos ticket issue

[#7]

No, actually step 1 is spnego which require ticket, https://hadoop.apache.org/docs/stable/hadoop-auth/Examples.html

What I don't understand is that before step 1, I can invoke hdfs command, but after step 1, I can not run step 1, seems my ticket is corrupted.

[#8]

Well, this is "interesting".

I think it's that specific realmless principal, "HTTP/sandbox.hortonworks.com@"; you don't have a TGT ticket for that empty realm, so fail. I've heard of this before

https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/terrors.html

Follow the instructions there; if it makes it go away, then it's a sign that the krb5 in the sandbox needs fixing

1. If you use kdestroy to delete the HTTP/sandbox.hortonworks.com@ ticket, what does that do?
2. download Kdiag and give it a run before and after the curl call: https://github.com/steveloughran/kdiag . `export HADOOP_JAAS_DEBUG=true` for extra info; grab stdout and stderr into a single file, and attach.
3. what does your /etc/krb5.conf say? Mine explicitly set dns_lookup_realm = false and dns_lookup_kdc = false
4. set the env vars and JVM properties covered in troubleshooting, see what's being negotiated. https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/Se...

[#9]

Not sure how to destroy this specify ticket, kdestroy will delete all the tickets. And I didn't add this ticket explicitly it is added after I invoke the the first curl command

my dns_lookup_realm = false and dns_lookup_kdc = false

attach the output of Kdiag

[#10 · Accepted solution]

The instruction on https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/terrors.html can fix this issue

Fix: add a domain_realm in /etc/krb5.conf mapping hostnames to realms

[domain_realm]

sandbox.hortonworks.com = EXAMPLE.COM