Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can not invoke hdfs command after invoking webhdfs operation in secure cluster

avatar
Super Collaborator

Here's what I did in my secure cluster:

1. Invokding webhdfs command: "curl -s -i --negotiate -u:anyUser http://sandbox.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS" (this works)

2. Invoking hdfs command: "hadoop fs -ls /"

But get the following error. (I need to kdestroy and kinti again to renew the ticket to make it work). This is weird, how can I make the webhdfs command not affect the hdfs command ?

16/02/23 03:38:41 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "sandbox.hortonworks.com/10.0.2.15"; destination host is: "sandbox.hortonworks.com":8020;

The only difference after step 1 in klist is that I have 2 extra HTTP principal, but I still have my principal jeff@EXAMPLE.COM

Default principal: jeff@EXAMPLE.COM Valid starting Expires Service principal 02/23/16 03:20:05 02/24/16 03:20:05 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 02/23/16 03:20:05 02/23/16 03:20:10 02/24/16 03:20:05 HTTP/sandbox.hortonworks.com@ renew until 02/23/16 03:20:05 02/23/16 03:20:10 02/24/16 03:20:05 HTTP/sandbox.hortonworks.com@EXAMPLE.COM renew until 02/23/16 03:20:05

1 ACCEPTED SOLUTION

avatar
Super Collaborator

The instruction on https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/terrors.html can fix this issue

Fix: add a domain_realm in /etc/krb5.conf mapping hostnames to realms

[domain_realm]

sandbox.hortonworks.com = EXAMPLE.COM

View solution in original post

9 REPLIES 9

avatar
Master Mentor
@jzhang

This is related to not having correct ticket from KDC. You can test by getting a ticket from hdfs keytab. kinit with hdfs headless and see if hdfs -ls works or not

avatar
Super Collaborator

Yeah, I see this should due to ticket issue. But I don't understand why the webhdfs command corrupt my ticket. If I only invoke step 2, everything is OK.

avatar
Master Mentor

@jzhang You will be running curl from the client node and hdfs from edge or master node.

avatar
Super Collaborator

No, I am running this in the same node. Actually I did it in sandbox. Only one node.

avatar
Master Mentor

@jzhang That's exactly my point that REST access will from the client so you should not worry about the kerberos ticket issue

avatar
Super Collaborator

No, actually step 1 is spnego which require ticket, https://hadoop.apache.org/docs/stable/hadoop-auth/Examples.html

What I don't understand is that before step 1, I can invoke hdfs command, but after step 1, I can not run step 1, seems my ticket is corrupted.

avatar

Well, this is "interesting".

I think it's that specific realmless principal, "HTTP/sandbox.hortonworks.com@"; you don't have a TGT ticket for that empty realm, so fail. I've heard of this before

https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/terrors.html

Follow the instructions there; if it makes it go away, then it's a sign that the krb5 in the sandbox needs fixing

  1. If you use kdestroy to delete the HTTP/sandbox.hortonworks.com@ ticket, what does that do?
  2. download Kdiag and give it a run before and after the curl call: https://github.com/steveloughran/kdiag . `export HADOOP_JAAS_DEBUG=true` for extra info; grab stdout and stderr into a single file, and attach.
  3. what does your /etc/krb5.conf say? Mine explicitly set dns_lookup_realm = false and dns_lookup_kdc = false
  4. set the env vars and JVM properties covered in troubleshooting, see what's being negotiated. https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/Se...

avatar
Super Collaborator

Not sure how to destroy this specify ticket, kdestroy will delete all the tickets. And I didn't add this ticket explicitly it is added after I invoke the the first curl command

my dns_lookup_realm = false and dns_lookup_kdc = false

attach the output of Kdiag

avatar
Super Collaborator

The instruction on https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/terrors.html can fix this issue

Fix: add a domain_realm in /etc/krb5.conf mapping hostnames to realms

[domain_realm]

sandbox.hortonworks.com = EXAMPLE.COM