Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

Highlighted

Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

Team,

I am using the

HDP 2.3.2 and Amabri 2.1.2,

FreeIPA as Kerberozation store,

HDFS located on the Isilon One FS 7.1.3.

Java Oracle JDK 1.8.101

Both Isilon and Hadoop Compute nodes point to same NTP server.

After enabling the Kerberozation in the Cluster I am getting the 'GSSException: Defective token detected (Mechanism level: AP_REP token id does not match!'. when starting the Job History Server,

Please refer the Java Stack trace here,

 2016-08-31 19:51:52,353 FATAL hs.JobHistoryServer (JobHistoryServer.java:launchJobHistoryServer(224)) - Error starting JobHistoryServer
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://schdp.bdl:8020/mr-history/done]
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:591)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:537)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:505)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:94)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:143)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:221)
        at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:231)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for mapred/phd19.bdl@BDL to schdp.bdl/11.11.11.192:8020; Host Details : local host is: "phd19.bdl/10.15.232.83"; destination host is: "schdp.bdl":8020;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
        at org.apache.hadoop.ipc.Client.call(Client.java:1431)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
        at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:130)
        at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1169)
        at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1165)
        at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
        at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1165)
        at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1630)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.mkdir(HistoryFileManager.java:644)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:574)
        ... 10 more
Caused by: java.io.IOException: Couldn't setup connection for mapred/phd19.bdl@BDL to schdp.bdl/11.11.11.192:8020
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:677)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:648)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:735)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
        at org.apache.hadoop.ipc.Client.call(Client.java:1397)
        ... 30 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Defective token detected (Mechanism level: AP_REP token id does not match!)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.hadoop.security.SaslRpcClient.saslEvaluateToken(SaslRpcClient.java:483)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:427)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
        ... 33 more
Caused by: GSSException: Defective token detected (Mechanism level: AP_REP token id does not match!)
        at sun.security.jgss.krb5.AcceptSecContextToken.<init>(AcceptSecContextToken.java:80)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:755)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
        ... 43 more


6 REPLIES 6
Highlighted

Re: Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

Have you reviewed the Isilon Kerberos posts listed here: https://community.emc.com/docs/DOC-39529

A lot of information is included on all the blog post about to successfully completing Isilon kerberos integration with hdp.

I would start their, but some high level areas to check:

DNS & Reverse DNS for all nodes, Isilon and SmartConnect nodes

SPN's are valid

Isilon zone is correctly kerberized

required ambari kerberos setting are correct

Highlighted

Re: Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

Recently dealt with this issue again, It looks like the issue is related to mismatches in the keytab versions between Isilon & KDC.

I would look at starting on page 27 of the following doc:

http://www.emc.com/collateral/TechnicalDocument/docu83576.pdf

Highlighted

Re: Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

@russ stevenson, That is a great post. Thanks for sharing.

Highlighted

Re: Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

You should make sure that there are no conflicts with the SPNEGO principal and keytab file - HTTP/<FQDN>@<REALM>. Isilon creates a SPNEGO principal for the host(s) it is installed on. Ambari will create a conflicting one when enabling Kerberos. If there is a conflict, bad things will happen... possibly, like the error you are seeing.

To search for conflicting service principal names, you can use the Active Directory tools; of, if you have the OpenLDAP clients installed on a Linux host, you can issue the following ldapsearch command:

ldapsearch -h ACTIVE_DIRECTORY_HOST -D BIND_USER -W -b BASE_DN '(servicePrincipalName=HTTP/ISILON_HOST_FQDN)' dn

Replacing:

  • ACTIVE_DIRECTORY_HOST with the hostname or IP address of your Active Directory (or LDAP server)
  • BIND_USER with the user principal or DN of a user with access to the Active Directory (or LDAP server)
  • BASE_DN with the DN of some container high in the LDAP tree to cover as may objects as possible
  • ISILON_HOST_FQDN with the fully qualified domain name of the Isilon host

If more than one entry is returned, you have a conflict and need to manually remove the one that was created by Ambari. This entry will have a CN like HTTP/ISILON_HOST_FQDN. Once removed, the issue should go away without any service restarts.

Highlighted

Re: Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

Highlighted

Re: Isilon - HDFS kerberos Error creating done directory: GSSException: Defective token detected

@russ stevenson, That is a great post. Thanks for sharing.

Don't have an account?
Coming from Hortonworks? Activate your account here