Issue with AMI with Parcel Preloaded

[#1]

I built an AMI using the preloaded-ami-builder script: 

https://github.com/cloudera/director-scripts/tree/master/preloaded-ami-builder

and my director config have the following specifying the parcel version I put in the AMI.:

products {
CDH: 5.5.1
}
parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.5.1/"]

But I keep getting this "CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels" exception when bootstrap-remote the config. Everything works if I remove the parcel_provisioner.sh in the packer though it takes time downloading the parcel. Please help me. I've spent days searching for solutions but no luck yet.

Another question not related to this issue but I see the builder has steps installing cloudera-manager-server and  cloudera-manager-daemons, why I am still seeing the following in the director provisioning step?

Installing cloudera-manager-server package

Installing cloudera-manager-daemons package

...

I suppose those should be skipped as well but please help me understand this.

Thanks!

-----------------------------------------------------------------------------------------------------------------------------------------------------

2016-01-19 17:50:03] INFO [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: >> EnableParcels/3 [CreateClusterContext{environment=Environment{name='platform Environment', provider=InstanceProvider ...
[2016-01-19 17:50:04] ERROR [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:77) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:36) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_65]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels$$EnhancerBySpringCGLIB$$7a951cb.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) [launchpad-pipeline-database-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-1.5.2.jar!/:1.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.util.BootstrapClusterUtils.getParcelProducts(BootstrapClusterUtils.java:319) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:57) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
... 33 common frames omitted

[#2]

This is strange -- truthfully, it looks like all your settings are copacetic, yet Director is failing to detect them here. A few more questions:

1. Could you try http://10.0.1.242:7180/api/v7/clusters/platform/parcels (the URL difference is v7 vs. v11).
2. So, the parcel that's being downloaded now is specifically for RHEL/CentOS 6. What base AMI are you currently using to build preloaded AMIs for CM/your cluster? Is it possible there's a mismatch, i.e. using RHEL/CentOS 7? If this is so, you may have to modify build-ami.sh to look for the parcel ending in el7 instead of el6.

As for multiple parcels -- this is doable, but the scripts as written do not support this, unfortunately. At the moment, you can either do this manually, or you can modify the packer scripts to download these other parcels. The latter shouldn't be too bad, but would require some packer familiarity.

I'm going to file a ticket internally for this feature, however, as it seems like a useful one.

[#3 · Accepted solution]

Thanks for your help! It finally works now!

Turns out that "normalizeInstance" has to set to true. It's a bit confusing though as in the comment it says the following and I thought most of them are already done in the preloaded AMI.

# Normalization includes:
# downloading and installing packages
# minimizing swappiness
# increasing the maximun number of open files
# mounting ephemeral disks
# resizing the root partition.

I will modify the packer scripts to download additional parcels we need. It will be great if you add support semi-preloeaded AMI so small parcels can be downloaded during provisioning.