Director bootstrap-remote fails while Importing Kerberos admin principal credential into new CM host

[#1]

Hello,

I'm trying to spawn cluster from Director deployer using bootstrap script, but it is failing while importing kerberos admin principal credentials. Please help me in resolving this.

Director bootstrap-remote execution output:

+ cloudera-director bootstrap-remote /tmp/tmp.ugzdsQEqpp.conf --lp.remote.username=admin --lp.remote.password=admin --lp.remote.hostAndPort=<director_FQDN>:7189
Process logs can be found at /root/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Cloudera Director 2.5.0 initializing ...
Connecting to http://<director_FQDN>:7189
Current user roles: [ROLE_READONLY, ROLE_ADMIN]
Configuration file passes all validation checks.
Creating a new environment...
Creating external database servers if configured...
Creating a new Cloudera Manager...
Creating a new CDH cluster...
* Requesting an instance for Cloudera Manager ............ done
* Installing screen package (1/1) ..... done
* Running bootstrap script #1 (crc32: 23bba9a9) ....... done
* Inspecting capabilities of <CM_IP_address> ..... done
* Normalizing 03135a68-963a-4eed-b599-32ef2e6ff7f4 ... done
* Installing ntp package (1/4) .... done
* Installing curl package (2/4) ..... done
* Installing nscd package (3/4) .... done
* Installing gdisk package (4/4) ................... done
* Resizing instance root partition ....... done
* Mounting all instance disk drives ........ done
* Waiting for new external database servers to start running .... done
* Installing repositories for Cloudera Manager ..... done
* Installing yum-utils package (1/3) .... done
* Installing cloudera-manager-daemons package (2/3) .... done
* Installing cloudera-manager-server package (3/3) .... done
* Installing mysql-connector-java package (1/2) .... done
* Installing mysql-devel package (2/2) .... done
* Installing krb5-workstation package (1/2) .... done
* Installing openldap-clients package (2/2) .... done
* Configuring external MYSQL database for Cloudera Manager ..... done
* Starting Cloudera Manager server ... done
* Waiting for Cloudera Manager server to start ... done
* Setting Cloudera Manager License ... done
* Applying Cloudera Manager license ... done
* Restarting Cloudera Manager ... done
* Waiting for Cloudera Manager server to restart ... done
* Configuring Cloudera Manager ... done
* Importing Kerberos admin principal credentials into Cloudera Manager ... done
* Suspended due to failure ...

ERROR from newly-launched CM agent logs:

POST /api/v7/cm/commands/importAdminCredentials?username=<admin_username>%40<REALM>&password=<admin_password>

2017-08-16 15:25:21,760 INFO 1599516774@scm-web-2:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ImportCredentials with sensitive arguments.

2017-08-16 15:25:26,990 ERROR CommandPusher:com.cloudera.cmf.command.CommandHelpers: ImportCredentials - Execution error:

java.io.IOException: /usr/share/cmf/bin/import_credentials.sh failed with exit code 1 and output of <<

ERROR from Director Logs:

[2017-08-17 11:48:49.747 -0400] ERROR [p-f34b78495a97-DefaultBootstrapDeploymentJob] POST /api/v9/environments/poc-test6-environment/deployments com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManager$ImportKrbAdminPrincipal - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.bootstrap.ApiCommandFailedException: Import of Kerberos admin principal credentials failed: /usr/share/cmf/bin/import_credentials.sh failed with exit code 1 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf539302729784501848.keytab
+ USER=<username>@<REALM>
+ PASSWD=REDACTED
+ KVNO=1
+ SLEEP=0
+ RHEL_FILE=/etc/redhat-release
+ '[' -f /etc/redhat-release ']'
+ set +e
+ grep Tikanga /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ grep 'CentOS release 5' /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ grep 'Scientific Linux release 5' /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ set -e
+ '[' -z /var/run/cloudera-scm-server/krb59606410535313402.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb59606410535313402.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb59606410535313402.conf
+ IFS=' '
+ read -a ENC_ARR
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p <username>@<REALM> -k 1 -e aes256-cts'
+ ktutil
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p <username>@<REALM> -k 1 -e aes128-cts'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p <username>@<REALM> -k 1 -e des3-hmac-sha1'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p <username>@<REALM> -k 1 -e arcfour-hmac'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p <username>@<REALM> -k 1 -e des-hmac-sha1'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p <username>@<REALM> -k 1 -e des-cbc-md5'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ for ENC in '"${ENC_ARR[@]}"'
+ echo 'addent -password -p <username>@<REALM> -k 1 -e des-cbc-crc'
+ '[' 0 -eq 1 ']'
+ echo REDACTED
+ echo 'wkt /var/run/cloudera-scm-server/cmf539302729784501848.keytab'
+ chmod 600 /var/run/cloudera-scm-server/cmf539302729784501848.keytab
+ kinit -k -t /var/run/cloudera-scm-server/cmf539302729784501848.keytab <username>@<REALM>
kinit: Cannot find KDC for realm "<REALM" while getting initial credentials

>>
at com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManager$ImportKrbAdminPrincipal.run(ConfigureClouderaManager.java:304)
at com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManager$ImportKrbAdminPrincipal.run(ConfigureClouderaManager.java:240)
at com.cloudera.launchpad.pipeline.job.Job6.runUnchecked(Job6.java:36)
at com.cloudera.launchpad.pipeline.job.Job6$$FastClassBySpringCGLIB$$54178506.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.GeneratedMethodAccessor394.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManager$ImportKrbAdminPrincipal$$EnhancerBySpringCGLIB$$27ccbc23.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:164)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2017-08-17 11:48:49.747 -0400] ERROR [p-f34b78495a97-DefaultBootstrapDeploymentJob] POST /api/v9/environments/poc-test6-environment/deployments com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManager$ImportKrbAdminPrincipal - c.c.l.p.DatabasePipelineRunner: Pipeline 4d7730cf-a142-4ed5-8728-f34b78495a97 suspended due to failure working on com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManager$ImportKrbAdminPrincipal
java.lang.RuntimeException: com.cloudera.launchpad.bootstrap.ApiCommandFailedException: Import of Kerberos admin principal credentials failed: /usr/share/cmf/bin/import_credentials.sh failed with exit code 1 and output of <<

Best.

Jagjeet

[#2]

Hi Jagjeet,

It looks like this line provides a hint to the problem:

kinit: Cannot find KDC for realm "<REALM" while getting initial credentials

You could start by verifying that you provided the right hostname / IP address for the KDC in your configuration, and that it is reachable over the network from the Cloudera Manager instance. Also check that the realm is spelled correctly and matches what the KDC supports.

Bill

[#3]

Hi Bill,

I noticed that CM successfully gets deployed with CM 5.11, but it failed with above exception if I use CM 5.12 version.
I found addition of 3 new config parameters in CM 5.12 UI under Administration > Settings > Kerberos:

KDC Admin Server Host -- Host where the KDC Admin server is located. Port number is optional and can be provided as hostname[:port]

Domain Name(s) -- Domain(s) which are mapped to this Kerberos Realm. This is used to generate [domain_realm] section. Also, the first domain is used as default_domain in [realms] section

Kerberos Trusted Realms -- List of Kerberos realms that all services on this Cloudera Manager should trust. This parameter is used to configure and verify krb5.conf file. The parameter is auto-configured while adding a peer, but it is recommended that users ensure the values are correct.

Q1: Can you let me know how I can define above parameters in config & properties script of Director deployer?
Q2: Does Director 2.5.0 supports addition of these parameters?

Best.

Jagjeet

[#4]

1) You can find the parameter names on https://www.cloudera.com/documentation/enterprise/properties/5-12-x/topics/cm_props_cmserver.html

2) The parameters are pass through in director conf script. As long as you add the entries in cloudera manager configs section, it will be passed to cloudera manager.

HTH

[#5]

Thank you for the link, I will try it out and confirm.

[#6 · Accepted solution]

Hi Everyone,

We're able to resolve issue by properly configuring Kerberos Safety Valve parameter. Earlier [domain_realms] section was not defined due to which cross-realms were getting defined under this section instead of [realms] section. After properly defining values under [domain_realms] section at last the above script worked. Thank you.

[#7]

To use aes encryption you need to install jce.

Be aware also that encryption types on cloudera side should be inline with kerberos server configuration.