GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)

[#1]

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Configuring-Hadoop-Securit...

After step 19,I restart the cluster,http://namenode:50070 required a username and password,and I use hdfs and it's password.

namenode log:

2014-06-26 17:55:39,907 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Authentication exception: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:360)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:349)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
at sun.security.jgss.GSSHeader.<init>(GSSHeader.java:97)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:306)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:327)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:309)
... 41 more

curl -v -u hdfs --negotiate http://namenode:50070 and press the password  worked.

What is the problem?

Is the username and password right?(I created the user and password by kadmin.local)?

rube

thx

[#2]

Did you deploy client configuration from cloudera manager?

What is in your krb5.conf?

What does klist -ef show after you kinit as your HDFS user?

What OS Distro & version are you on?

[#3]

Thank you for your reply!

1.I did from CM

2.krb5.conf

 ....log conf....

[libdefaults]
default_realm = HADOOP.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true

[realms]
HADOOP.COM = {
kdc = datanode14.yeahmobi.com
admin_server = datanode14.yeahmobi.com
}

[domain_realm]
.yeahmobi.com = HADOOP.COM
namenode11 = HADOOP.COM
datanode14 = HADOOP.COM
datanode12 = HADOOP.COM
datanode13 = HADOOP.COM

3.hdfs klist -ef

Default principal: hdfs@HADOOP.COM

Valid starting Expires Service principal
06/26/14 16:31:27 06/27/14 16:31:27 krbtgt/HADOOP.COM@HADOOP.COM
renew until 07/03/14 16:31:27, Flags: FRI
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
06/26/14 16:31:36 06/27/14 16:31:27 HTTP/namenode11.yeahmobi.com@HADOOP.COM
renew until 07/01/14 16:31:36, Flags: FRT
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

4.centos6.4

[#4]

So for the [domain_realm] section,  you can focus on the domain mapping to the realm, so 

[domain_realm]
.yeahmobi.com = HADOOP.COM

yeahmobi.com = HADOOP.COM

So when you read that above, it is stating

.yeahmobi.com = HADOOP.COM would handle any_subdomain.yeahmobi.com being mapped to the realm HADOOP.COM

yeahmobi.com = HADOOP.COM would handle any_hostname.yeahmobi.com being mapped to the realm HADOOP.COM

The host name only references in your [domain_realms] section are not valid.  

Make sure you have deployed the JCE policy files for the version of JDK you are using in the cluster.  That is indicating your kerberos configuration is using AES-256 keys which are a stong encryption form of key.  The default JDK does not have those strength ciphers available by default. The jar files get copied into your /usr/java/jdk1.*/jre/lib/security path, replacing the existing ones.  Restart services to have the JVM come up ready to use the strong ecnryption (aes-256) ciphers.

You can obtain the proper JDK version's JCE policy files here:

JDK 1.6 http://www.oracle.com/technetwork/java/javase/downloads/jce-6-download-429243.html
JDK 1.7 http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html

[#5]

I had deployed JCE.
It does not work.The cluster has 4 nodes,hosts:
172.20.0.11 namenode11.yeahmobi.com namenode11
172.20.0.12 datanode12.yeahmobi.com datanode12
172.20.0.13 datanode13.yeahmobi.com datanode13
172.20.0.14 datanode14.yeahmobi.com datanode14
I guess,maybe I missed some configurations.
I had Enable Authentication for HTTP Web-Consoles,if want to access webUI(eg:namenode:50070) from a windows client,what should I do?
Should I do Integrating Hadoop Security with Alternate Authentication?
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Security-Guide/cdh5s...