Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Solved Go to solution

*Closed* - Cloudbreak on Azure: Kerberize a cluster against Active Directory - error related to length of CN

Labels:
avatar
author-rank Indra
Contributor

Created ‎11-29-2017 05:15 AM

Provisioned a cluster on Azure using Cloudbreak and then...

Attempted:
Kerberize the cluster using Ambari Kerberos automatic wizard, against an existing Active Directory prepped ahead of time

Issue:
The kerberos set up fails when it tries to create a SPN for zookeeper. The error seems to point to length of CN exceeding max length limit.

STDERR from Ambari Kerberos wizard UI:

2017-11-28 16:41:58,340 - Failed to create principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM - 
Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM

STDOUT from Ambari Kerberos wizard UI:

2017-11-28 16:41:57,944 - Processing identities...
2017-11-28 16:41:58,019 - Processing principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,021 - Principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,048 - Processing principal, ambari-qa-denali@DENALI.COM
2017-11-28 16:41:58,049 - Principal, ambari-qa-denali@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,076 - Processing principal, hdfs-denali@DENALI.COM
2017-11-28 16:41:58,077 - Principal, hdfs-denali@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,104 - Processing principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,106 - Principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,133 - Processing principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,134 - Principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,163 - Processing principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,165 - Principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,193 - Processing principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,195 - Principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,221 - Processing principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,222 - Principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,248 - Processing principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,249 - Principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,276 - Processing principal, jn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,278 - Principal, jn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,306 - Processing principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,307 - Principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,334 - Processing principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM

Just to show that several SPs got created, it consistently fails at zookeeper.

Troubleshooting attempted:
Reduced zookeeper to zk, got past the error, only to fail for amshbase, reduced this to amshb, got past the setup.
Failed during smoke testing; We cannot be changing service principal names, this was merely to test the hypothesis that it was length related.

Ambari log:

29 Nov 2017 00:47:08,143  INFO [Server Action Executor Worker 464] StackAdvisorRunner:71 -     advisor script stderr:
29 Nov 2017 00:47:08,152  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SQOOP=[SQOOP] to auth to local mapping
29 Nov 2017 00:47:08,152  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SQOOP to auth to local mapping
29 Nov 2017 00:47:08,152  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service HDFS=[HDFS_CLIENT, ZKFC, DATANODE, JOURNALNODE, NAMENODE] to auth to local mapping
29 Nov 2017 00:47:08,152  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HDFS_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component DATANODE to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component JOURNALNODE to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component NAMENODE to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service TEZ=[TEZ_CLIENT] to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component TEZ_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service MAPREDUCE2=[MAPREDUCE2_CLIENT, HISTORYSERVER] to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HISTORYSERVER to auth to local mapping
29 Nov 2017 00:47:08,153  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service ZOOKEEPER=[ZOOKEEPER_SERVER, ZOOKEEPER_CLIENT] to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component ZOOKEEPER_SERVER to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service YARN=[NODEMANAGER, YARN_CLIENT, APP_TIMELINE_SERVER, RESOURCEMANAGER] to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component NODEMANAGER to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component APP_TIMELINE_SERVER to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component RESOURCEMANAGER to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service KERBEROS=[KERBEROS_CLIENT] to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component KERBEROS_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service PIG=[PIG] to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component PIG to auth to local mapping
29 Nov 2017 00:47:08,154  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service HIVE=[HIVE_SERVER, MYSQL_SERVER, HIVE_METASTORE, HIVE_CLIENT, WEBHCAT_SERVER] to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HIVE_SERVER to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HIVE_METASTORE to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component WEBHCAT_SERVER to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SLIDER=[SLIDER] to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SLIDER to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service AMBARI_METRICS=[METRICS_MONITOR, METRICS_COLLECTOR] to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component METRICS_COLLECTOR to auth to local mapping
29 Nov 2017 00:47:08,155  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SMARTSENSE=[HST_AGENT, HST_SERVER] to auth to local mapping
29 Nov 2017 00:47:08,156  INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SPARK2=[SPARK2_CLIENT, SPARK2_JOBHISTORYSERVER] to auth to local mapping
29 Nov 2017 00:47:08,156  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SPARK2_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,156  INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SPARK2_JOBHISTORYSERVER to auth to local mapping
29 Nov 2017 00:47:08,557  INFO [Server Action Executor Worker 465] KerberosServerAction:353 - Processing identities...
29 Nov 2017 00:47:08,629  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,657  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hdfs-denali@DENALI.COM
29 Nov 2017 00:47:08,684  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,713  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,740  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,768  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,796  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,824  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,852  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,879  INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,885 ERROR [Server Action Executor Worker 465] CreatePrincipalsServerAction:297 - Failed to create principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM - Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createPrincipal(ADKerberosOperationHandler.java:331)
        at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.createPrincipal(CreatePrincipalsServerAction.java:256)
        at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.processIdentity(CreatePrincipalsServerAction.java:159)
        at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:532)
        at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:414)
        at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.execute(CreatePrincipalsServerAction.java:91)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:555)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:492)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 19 - 00002082: AtrErr: DSID-031519A3, #1:
        0: 00002082: DSID-031519A3, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 3 (cn):len 138
 ]; remaining name '"cn=zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net,OU=hdpou,DC=denali,DC=com"'
        at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3149)
        at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3082)
        at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2888)
        at com.sun.jndi.ldap.LdapCtx.c_createSubcontext(LdapCtx.java:812)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_createSubcontext(ComponentDirContext.java:341)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.createSubcontext(PartialCompositeDirContext.java:268)
        at javax.naming.directory.InitialDirContext.createSubcontext(InitialDirContext.java:202)
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createPrincipal(ADKerberosOperationHandler.java:329)
        ... 8 more
29 Nov 2017 00:47:08,886  INFO [Server Action Executor Worker 465] KerberosServerAction:457 - Processing identities completed.
29 Nov 2017 00:47:09,559 ERROR [ambari-action-scheduler] ActionScheduler:440 - Operation completely failed, aborting request id: 39
29 Nov 2017 00:47:09,560  INFO [ambari-action-scheduler] ActionScheduler:952 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION
29 Nov 2017 00:47:09,585  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 466 stageId 2
29 Nov 2017 00:47:09,585  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 467 stageId 3
29 Nov 2017 00:47:09,585  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-e0.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 468 stageId 4
29 Nov 2017 00:47:09,585  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m1.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 469 stageId 4
29 Nov 2017 00:47:09,585  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m12.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 470 stageId 4
29 Nov 2017 00:47:09,586  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 471 stageId 4
29 Nov 2017 00:47:09,586  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m34.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 472 stageId 4
29 Nov 2017 00:47:09,586  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s15.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 473 stageId 4
29 Nov 2017 00:47:09,586  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 474 stageId 4
29 Nov 2017 00:47:09,586  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s17.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 475 stageId 4
29 Nov 2017 00:47:09,586  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 476 stageId 5
29 Nov 2017 00:47:09,586  INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 477 stageId 6
29 Nov 2017 00:48:41,263  INFO [pool-18-thread-1] MetricsServiceImpl:64 - Checking for metrics sink initialization


Deduction:
The length is beyond the limit acceptable by Active Directory
OK:
yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM

FAILS:
zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
amshbase/den-m1.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM

Question:

(1) Anyone run into this issue that has a solution to share? I know I can pop a MIT Kerberos KDC in front of AD...looking for options.

(2) Does the Cloudbreak team have any guidance?

Thanks in advance.
I am now attempting to provision via Cloudbreak - kerberize at provision-time against existing Active Directory. Fingers crossed.

1,511 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank Indra
Contributor

Created ‎12-03-2017 06:59 PM

Solution:
VM FQDN needs to be shorter than what you get with Azure defaults. This is not a Cloudbreak issue.

View solution in original post

1,315 Views
0 Kudos
3 REPLIES 3

avatar
author-rank Indra
Contributor

Created ‎12-02-2017 06:55 AM

Attempting to create a HDP cluster with Kerberos at provision time against AD failed.
Issue is tied to the same as one reported - very long VM FQDN - exceeding upper limits defined in AD, AAD DS

1,315 Views
0 Kudos

avatar
author-rank Indra
Contributor

Created ‎12-03-2017 06:59 PM

Solution:
VM FQDN needs to be shorter than what you get with Azure defaults. This is not a Cloudbreak issue.

1,316 Views
0 Kudos

avatar
author-rank james_bashforth
Explorer

Created ‎08-02-2018 02:40 AM

I am having the same issue with a kerberized cluster created through cloudbreak 2.7. Did you manage to find a workaround the fqdn length?

1,315 Views
0 Kudos
Announcements
Product Announcements
[ANNOUNCE] Cloudera on Public Cloud Runtime 7.2.18 is GA!
What's New @ Cloudera
Cloudera Operational Database (COD) supports enhancements to...
Community Announcements
March 2024 Community Highlights
Community Announcements
Celebrating 100,000 Members: A Journey of Growth and Connect...
Product Announcements
[ANNOUNCE] New Cloudera JDBC Connector for Apache Hive 2.6.2...
View More Announcements