question Can not invoke hdfs command after invoking webhdfs operation in secure cluster in Archives of Support Questions (Read Only)

Can not invoke hdfs command after invoking webhdfs operation in secure cluster

jzhang — Tue, 23 Feb 2016 11:42:17 GMT

Here's what I did in my secure cluster:

1. Invokding webhdfs command: "curl -s -i --negotiate -u:anyUser http://sandbox.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS" (this works)

2. Invoking hdfs command: "hadoop fs -ls /"

But get the following error. (I need to kdestroy and kinti again to renew the ticket to make it work). This is weird, how can I make the webhdfs command not affect the hdfs command ?

16/02/23 03:38:41 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "sandbox.hortonworks.com/10.0.2.15"; destination host is: "sandbox.hortonworks.com":8020;

The only difference after step 1 in klist is that I have 2 extra HTTP principal, but I still have my principal jeff@EXAMPLE.COM

Default principal: jeff@EXAMPLE.COM Valid starting Expires Service principal 02/23/16 03:20:05 02/24/16 03:20:05 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 02/23/16 03:20:05 02/23/16 03:20:10 02/24/16 03:20:05 HTTP/sandbox.hortonworks.com@ renew until 02/23/16 03:20:05 02/23/16 03:20:10 02/24/16 03:20:05 HTTP/sandbox.hortonworks.com@EXAMPLE.COM renew until 02/23/16 03:20:05

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

nsabharwal — Tue, 23 Feb 2016 14:06:25 GMT

@jzhang

This is related to not having correct ticket from KDC. You can test by getting a ticket from hdfs keytab. kinit with hdfs headless and see if hdfs -ls works or not

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

jzhang — Tue, 23 Feb 2016 14:08:08 GMT

Yeah, I see this should due to ticket issue. But I don't understand why the webhdfs command corrupt my ticket. If I only invoke step 2, everything is OK.

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

nsabharwal — Tue, 23 Feb 2016 14:20:07 GMT

@jzhang You will be running curl from the client node and hdfs from edge or master node.

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

jzhang — Tue, 23 Feb 2016 14:29:07 GMT

No, I am running this in the same node. Actually I did it in sandbox. Only one node.

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

nsabharwal — Tue, 23 Feb 2016 17:47:06 GMT

@jzhang That's exactly my point that REST access will from the client so you should not worry about the kerberos ticket issue

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

jzhang — Wed, 24 Feb 2016 17:54:44 GMT

No, actually step 1 is spnego which require ticket, https://hadoop.apache.org/docs/stable/hadoop-auth/Examples.html

What I don't understand is that before step 1, I can invoke hdfs command, but after step 1, I can not run step 1, seems my ticket is corrupted.

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

stevel — Thu, 03 Mar 2016 19:44:17 GMT

Well, this is "interesting".

I think it's that specific realmless principal, "HTTP/sandbox.hortonworks.com@"; you don't have a TGT ticket for that empty realm, so fail. I've heard of this before

https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/terrors.html

Follow the instructions there; if it makes it go away, then it's a sign that the krb5 in the sandbox needs fixing

If you use kdestroy to delete the HTTP/sandbox.hortonworks.com@ ticket, what does that do?
download Kdiag and give it a run before and after the curl call: https://github.com/steveloughran/kdiag . `export HADOOP_JAAS_DEBUG=true` for extra info; grab stdout and stderr into a single file, and attach.
what does your /etc/krb5.conf say? Mine explicitly set dns_lookup_realm = false and dns_lookup_kdc = false
set the env vars and JVM properties covered in troubleshooting, see what's being negotiated. https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

jzhang — Thu, 03 Mar 2016 21:27:59 GMT

Not sure how to destroy this specify ticket, kdestroy will delete all the tickets. And I didn't add this ticket explicitly it is added after I invoke the the first curl command

my dns_lookup_realm = false and dns_lookup_kdc = false

attach the output of Kdiag

Re: Can not invoke hdfs command after invoking webhdfs operation in secure cluster

jzhang — Thu, 03 Mar 2016 22:44:13 GMT

The instruction on https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/terrors.html can fix this issue

Fix: add a domain_realm in /etc/krb5.conf mapping hostnames to realms

[domain_realm]

sandbox.hortonworks.com = EXAMPLE.COM