Provisioned a cluster on Azure using Cloudbreak and then...
Attempted:
Kerberize the cluster using Ambari Kerberos automatic wizard, against an existing Active Directory prepped ahead of time
Issue:
The kerberos set up fails when it tries to create a SPN for zookeeper. The error seems to point to length of CN exceeding max length limit.
STDERR from Ambari Kerberos wizard UI:
2017-11-28 16:41:58,340 - Failed to create principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM -
Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
STDOUT from Ambari Kerberos wizard UI:
2017-11-28 16:41:57,944 - Processing identities...
2017-11-28 16:41:58,019 - Processing principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,021 - Principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,048 - Processing principal, [email protected]
2017-11-28 16:41:58,049 - Principal, [email protected], already exists, setting new password
2017-11-28 16:41:58,076 - Processing principal, [email protected]
2017-11-28 16:41:58,077 - Principal, [email protected], already exists, setting new password
2017-11-28 16:41:58,104 - Processing principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,106 - Principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,133 - Processing principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,134 - Principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,163 - Processing principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,165 - Principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,193 - Processing principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,195 - Principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,221 - Processing principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,222 - Principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,248 - Processing principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,249 - Principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,276 - Processing principal, jn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,278 - Principal, jn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,306 - Processing principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
2017-11-28 16:41:58,307 - Principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM, already exists, setting new password
2017-11-28 16:41:58,334 - Processing principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
Just to show that several SPs got created, it consistently fails at zookeeper.
Troubleshooting attempted:
Reduced zookeeper to zk, got past the error, only to fail for amshbase, reduced this to amshb, got past the setup.
Failed during smoke testing; We cannot be changing service principal names, this was merely to test the hypothesis that it was length related.
Ambari log:
29 Nov 2017 00:47:08,143 INFO [Server Action Executor Worker 464] StackAdvisorRunner:71 - advisor script stderr:
29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SQOOP=[SQOOP] to auth to local mapping
29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SQOOP to auth to local mapping
29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service HDFS=[HDFS_CLIENT, ZKFC, DATANODE, JOURNALNODE, NAMENODE] to auth to local mapping
29 Nov 2017 00:47:08,152 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HDFS_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component DATANODE to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component JOURNALNODE to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component NAMENODE to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service TEZ=[TEZ_CLIENT] to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component TEZ_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service MAPREDUCE2=[MAPREDUCE2_CLIENT, HISTORYSERVER] to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HISTORYSERVER to auth to local mapping
29 Nov 2017 00:47:08,153 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service ZOOKEEPER=[ZOOKEEPER_SERVER, ZOOKEEPER_CLIENT] to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component ZOOKEEPER_SERVER to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service YARN=[NODEMANAGER, YARN_CLIENT, APP_TIMELINE_SERVER, RESOURCEMANAGER] to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component NODEMANAGER to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component APP_TIMELINE_SERVER to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component RESOURCEMANAGER to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service KERBEROS=[KERBEROS_CLIENT] to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component KERBEROS_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service PIG=[PIG] to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component PIG to auth to local mapping
29 Nov 2017 00:47:08,154 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service HIVE=[HIVE_SERVER, MYSQL_SERVER, HIVE_METASTORE, HIVE_CLIENT, WEBHCAT_SERVER] to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HIVE_SERVER to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component HIVE_METASTORE to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component WEBHCAT_SERVER to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SLIDER=[SLIDER] to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SLIDER to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service AMBARI_METRICS=[METRICS_MONITOR, METRICS_COLLECTOR] to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component METRICS_COLLECTOR to auth to local mapping
29 Nov 2017 00:47:08,155 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SMARTSENSE=[HST_AGENT, HST_SERVER] to auth to local mapping
29 Nov 2017 00:47:08,156 INFO [Server Action Executor Worker 464] KerberosHelperImpl:950 - Adding identities for service SPARK2=[SPARK2_CLIENT, SPARK2_JOBHISTORYSERVER] to auth to local mapping
29 Nov 2017 00:47:08,156 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SPARK2_CLIENT to auth to local mapping
29 Nov 2017 00:47:08,156 INFO [Server Action Executor Worker 464] KerberosHelperImpl:967 - Adding identities for component SPARK2_JOBHISTORYSERVER to auth to local mapping
29 Nov 2017 00:47:08,557 INFO [Server Action Executor Worker 465] KerberosServerAction:353 - Processing identities...
29 Nov 2017 00:47:08,629 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, HTTP/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,657 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, [email protected]
29 Nov 2017 00:47:08,684 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, dn/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,713 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, nm/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,740 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hive/den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,768 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, HTTP/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,796 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,824 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, hive/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,852 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, rm/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,879 INFO [Server Action Executor Worker 465] CreatePrincipalsServerAction:203 - Processing principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
29 Nov 2017 00:47:08,885 ERROR [Server Action Executor Worker 465] CreatePrincipalsServerAction:297 - Failed to create principal, zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM - Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Can not create principal : zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createPrincipal(ADKerberosOperationHandler.java:331)
at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.createPrincipal(CreatePrincipalsServerAction.java:256)
at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.processIdentity(CreatePrincipalsServerAction.java:159)
at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:532)
at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:414)
at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.execute(CreatePrincipalsServerAction.java:91)
at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:555)
at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:492)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 19 - 00002082: AtrErr: DSID-031519A3, #1:
0: 00002082: DSID-031519A3, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 3 (cn):len 138
]; remaining name '"cn=zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net,OU=hdpou,DC=denali,DC=com"'
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3149)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3082)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2888)
at com.sun.jndi.ldap.LdapCtx.c_createSubcontext(LdapCtx.java:812)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_createSubcontext(ComponentDirContext.java:341)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.createSubcontext(PartialCompositeDirContext.java:268)
at javax.naming.directory.InitialDirContext.createSubcontext(InitialDirContext.java:202)
at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createPrincipal(ADKerberosOperationHandler.java:329)
... 8 more
29 Nov 2017 00:47:08,886 INFO [Server Action Executor Worker 465] KerberosServerAction:457 - Processing identities completed.
29 Nov 2017 00:47:09,559 ERROR [ambari-action-scheduler] ActionScheduler:440 - Operation completely failed, aborting request id: 39
29 Nov 2017 00:47:09,560 INFO [ambari-action-scheduler] ActionScheduler:952 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION
29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 466 stageId 2
29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 467 stageId 3
29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-e0.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 468 stageId 4
29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m1.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 469 stageId 4
29 Nov 2017 00:47:09,585 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m12.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 470 stageId 4
29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 471 stageId 4
29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-m34.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 472 stageId 4
29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s15.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 473 stageId 4
29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s16.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 474 stageId 4
29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname den-s17.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net role KERBEROS_CLIENT requestId 39 taskId 475 stageId 4
29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 476 stageId 5
29 Nov 2017 00:47:09,586 INFO [ambari-action-scheduler] ActionDBAccessorImpl:218 - Aborting command. Hostname null role AMBARI_SERVER_ACTION requestId 39 taskId 477 stageId 6
29 Nov 2017 00:48:41,263 INFO [pool-18-thread-1] MetricsServiceImpl:64 - Checking for metrics sink initialization
Deduction:
The length is beyond the limit acceptable by Active Directory
OK:
yarn/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
FAILS:
zookeeper/den-m23.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
amshbase/den-m1.rxo2hisyweyefnkiphzw3u2whg.cx.internal.cloudapp.net@DENALI.COM
Question:
(1) Anyone run into this issue that has a solution to share? I know I can pop a MIT Kerberos KDC in front of AD...looking for options.
(2) Does the Cloudbreak team have any guidance?
Thanks in advance.
I am now attempting to provision via Cloudbreak - kerberize at provision-time against existing Active Directory. Fingers crossed.
