Cloudera Director hangs Enabling Kerberos

[#1]

Performing a test install in AWS and all is well until:

...

Cloudera Manager ready.
Creating cluster pd472_cdh57_impala_cert ...
* Starting ..... done
* Requesting 7 instance(s) in 3 group(s) ............................................. done
* Preparing instances in parallel (20 at a time) ................................................................................... done
* Waiting for Cloudera Manager installation to complete ... done
* Installing Cloudera Manager agents on all instances in parallel (20 at a time) .......... done
* Creating CDH5 cluster using the new instances ... done
* Creating cluster: pd472_cdh57_impala_cert ... done
* Downloading parcels: CDH-5.7.2-1.cdh5.7.2.p0.18 .... done
* Distributing parcels: CDH-5.7.2-1.cdh5.7.2.p0.18 ... done
* Switching parcel distribution rate limits back to defaults: 51200KB/s with 25 concurrent uploads ... done
* Activating parcels: CDH-5.7.2-1.cdh5.7.2.p0.18 ........ done
* Enabling Kerberos ......

At this point the log is repeating the following:

[2016-08-25 22:40:49] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
java.lang.RuntimeException: com.cloudera.launchpad.bootstrap.ClouderaManagerCommandFailedException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-15.0.jar!/:na]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:69) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0]
at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.ConfigureClusterForKerberos$DeployClientConfigs$$EnhancerBySpringCGLIB$$c9aa228d.runUnchecked(<generated>) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.1.0.jar!/:2.1.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.1.0.jar!/:2.1.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.1.0.jar!/:2.1.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.1.0.jar!/:2.1.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.1.0.jar!/:2.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: com.cloudera.launchpad.bootstrap.ClouderaManagerCommandFailedException: null
at com.cloudera.launchpad.bootstrap.cluster.ConfigureClusterForKerberos$DeployClientConfigs.run(ConfigureClusterForKerberos.java:146) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0]
at com.cloudera.launchpad.bootstrap.cluster.ConfigureClusterForKerberos$DeployClientConfigs.run(ConfigureClusterForKerberos.java:122) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0]
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0]
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0]
... 25 common frames omitted

Any indication as to what could be the source of the error? Is it the template, the KDC, connectivity? My best guess is that the KDC setup isn't supporting the request being made but the error appears to be incomplete or just not useful.

[#2 · Accepted solution]

Hi ProKirk,

Looks like our logging could be better here. The exception is being thrown because Director asked Cloudera Manager to deploy updated client configurations after Kerberos was configured, but that deployment failed. I recommend taking a look at the Cloudera Manager instance that Director spun up to see what went wrong. Director doesn't have much visibility into the details of failed Cloudera Manager commands on its side.

If you log in to Cloudera Manager, you can see its recent command history by selecting the scroll icon on the upper right, and then pressing the "All Recent Commands" button. You can also look in /var/log/cloudera-scm-server, or select Diagnostics > Logs or Diagnostics > Server Log from the Cloudera Manager navigation bar, to look at the server log directly.

Let us know what you find! Hopefully there will be good information on what went wrong.

[#3]

Thanks for the response. In the end I had to log into CM and fix critical erorrs for the Impala service. No daemon roles were assigned via the config file. The fix prompted the Enable Kerberos step to complete successfully.

After that I got another error that kept Imapala from starting (short-circuit read access) but that is another problem for another post.

Thanks again