Support Questions
Find answers, ask questions, and share your expertise

Integrating Kerberos with AD using Ambari

Integrating Kerberos with AD using Ambari

Running into the following error when enter all KDC and Kadmin details to go to next screen, where the error comes up when 'Testing the KDC client'

Any inputs / pointers are appreciated.

10 Feb 2016 10:33:53,818 ERROR [Server Action Executor Worker 1663] CreatePrincipalsServerAction:199 - Failed to create principal, hadoop_dev-021016@EXAMPLE.com - can not check if principal exists: hadoop_dev-021016@EXAMPLE.com
org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: can not check if principal exists: hadoop_dev-021016@EXAMPLE.com
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.principalExists(ADKerberosOperationHandler.java:223)
        at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.processIdentity(CreatePrincipalsServerAction.java:155)
        at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:512)
        at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:401)
        at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.execute(CreatePrincipalsServerAction.java:79)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:537)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:474)
        at java.lang.Thread.run(Thread.java:744)
Caused by: javax.naming.LimitExceededException: Referral limit exceeded [Root exception is com.sun.jndi.ldap.LdapReferralException: [LDAP: error code 10 - 0000202B: RefErr: DSID-031007F3, data 0, 1 access points
        ref 1: 'example.com'
^@]; remaining name '']; remaining name ''
        at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2938)
        at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1849)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:657)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1867)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:276)
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.findPrincipalDN(ADKerberosOperationHandler.java:559)
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.principalExists(ADKerberosOperationHandler.java:221)
        ... 7 more
Caused by: com.sun.jndi.ldap.LdapReferralException: [LDAP: error code 10 - 0000202B: RefErr: DSID-031007F3, data 0, 1 access points
        ref 1: 'example.com'
^@]; remaining name ''
        at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2927)
        ... 65 more
10 Feb 2016 10:33:53,822  INFO [Server Action Executor Worker 1663] KerberosServerAction:444 - Processing identities completed.
10 Feb 2016 10:33:54,335  WARN [ambari-action-scheduler] ActionScheduler:317 - Operation completely failed, aborting request id: 199
10 Feb 2016 10:33:54,335  INFO [ambari-action-scheduler] ActionScheduler:699 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION
10 Feb 2016 10:33:54,339  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodemn02.example.com role AMBARI_SERVER_ACTION requestId null taskId 1664 stageId null
10 Feb 2016 10:33:54,339  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn05.example.com role KERBEROS_CLIENT requestId null taskId 1665 stageId null
10 Feb 2016 10:33:54,340  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn06.example.com role KERBEROS_CLIENT requestId null taskId 1666 stageId null
10 Feb 2016 10:33:54,340  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn07.example.com role KERBEROS_CLIENT requestId null taskId 1667 stageId null
10 Feb 2016 10:33:54,340  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn08.example.com role KERBEROS_CLIENT requestId null taskId 1668 stageId null
10 Feb 2016 10:33:54,340  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodemn02.example.com role KERBEROS_CLIENT requestId null taskId 1669 stageId null
10 Feb 2016 10:33:54,340  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodesn02.example.com role KERBEROS_CLIENT requestId null taskId 1670 stageId null
10 Feb 2016 10:33:54,341  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodemn02.example.com role AMBARI_SERVER_ACTION requestId null taskId 1671 stageId null
10 Feb 2016 10:33:54,341  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn05.example.com role KERBEROS_SERVICE_CHECK requestId null taskId 1672 stageId null
10 Feb 2016 10:33:54,341  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodemn02.example.com role AMBARI_SERVER_ACTION requestId null taskId 1673 stageId null
10 Feb 2016 10:33:54,341  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodemn02.example.com role AMBARI_SERVER_ACTION requestId null taskId 1674 stageId null
10 Feb 2016 10:33:54,342  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn05.example.com role KERBEROS_CLIENT requestId null taskId 1675 stageId null
10 Feb 2016 10:33:54,342  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn06.example.com role KERBEROS_CLIENT requestId null taskId 1676 stageId null
10 Feb 2016 10:33:54,342  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn07.example.com role KERBEROS_CLIENT requestId null taskId 1677 stageId null
10 Feb 2016 10:33:54,342  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodedn08.example.com role KERBEROS_CLIENT requestId null taskId 1678 stageId null
10 Feb 2016 10:33:54,342  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodemn02.example.com role KERBEROS_CLIENT requestId null taskId 1679 stageId null
10 Feb 2016 10:33:54,343  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodesn02.example.com role KERBEROS_CLIENT requestId null taskId 1680 stageId null
10 Feb 2016 10:33:54,343  INFO [ambari-action-scheduler] ActionDBAccessorImpl:186 - Aborting command. Hostname hdpnodemn02.example.com role AMBARI_SERVER_ACTION requestId null taskId 1681 stageId null
10 Feb 2016 10:34:10,173  INFO [qtp-client-182831] PersistKeyValueService:82 - Looking for keyName hostPopup-pagination-displayLength-admin
~

7 REPLIES 7

Re: Integrating Kerberos with AD using Ambari

@Paul Codding

Help !! Thanks!

Re: Integrating Kerberos with AD using Ambari

You should check the realm name you used when enabling Kerberos. It appears there may be an issue since the realm name is really expected to be in all capital letters.

Looking at

hadoop_dev-021016@EXAMPLE.com

This seems like an issue, I would expect this to be

hadoop_dev-021016@EXAMPLE.COM

If this isn't the issue, check the container DN that was specified to make sure that it exists and that the administrator used to access the Active Directory can access that container.

Re: Integrating Kerberos with AD using Ambari

That checks out.. I changed (search - replace all) the actual customer realm in the log before pasting here and accidentally left the .com in lower case. I checked the original log file and the realm is correct.

Re: Integrating Kerberos with AD using Ambari

If all else checks out, it seems like there may be a container DN issue. I would make sure the administrator credentials allow for writes to the container. Also, make sure LDAPS is being used.. else Ambari will fail setting the account's password.

Re: Integrating Kerberos with AD using Ambari

Does it mean that i cannot install Kerberos with ambari using LDAP, it needs to be LDAPS? I think i am having this issue 'Ambari will fail setting the account's password'

Re: Integrating Kerberos with AD using Ambari

@Saurabh Singh

Correct.. you must use LDAPS else Ambari will fail to set/update passwords for the accounts it manages. This is a restriction imposed by Active Directory in order to protect the password from being exposed as plaintext.

Re: Integrating Kerberos with AD using Ambari