Issue with AMI with Parcel Preloaded

[#1]

I built an AMI using the preloaded-ami-builder script: 

https://github.com/cloudera/director-scripts/tree/master/preloaded-ami-builder

and my director config have the following specifying the parcel version I put in the AMI.:

products {
CDH: 5.5.1
}
parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.5.1/"]

But I keep getting this "CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels" exception when bootstrap-remote the config. Everything works if I remove the parcel_provisioner.sh in the packer though it takes time downloading the parcel. Please help me. I've spent days searching for solutions but no luck yet.

Another question not related to this issue but I see the builder has steps installing cloudera-manager-server and  cloudera-manager-daemons, why I am still seeing the following in the director provisioning step?

Installing cloudera-manager-server package

Installing cloudera-manager-daemons package

...

I suppose those should be skipped as well but please help me understand this.

Thanks!

-----------------------------------------------------------------------------------------------------------------------------------------------------

2016-01-19 17:50:03] INFO [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: >> EnableParcels/3 [CreateClusterContext{environment=Environment{name='platform Environment', provider=InstanceProvider ...
[2016-01-19 17:50:04] ERROR [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:77) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:36) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_65]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels$$EnhancerBySpringCGLIB$$7a951cb.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) [launchpad-pipeline-database-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-1.5.2.jar!/:1.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.util.BootstrapClusterUtils.getParcelProducts(BootstrapClusterUtils.java:319) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:57) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
... 33 common frames omitted

[#2]

What versions of Cloudera Director and Cloudera Manager are you using?

You may want to take a look at this comment for possible info on problems with version incompatibilities: http://community.cloudera.com/t5/Cloudera-Director-Cloud-based/Cloudera-Director-bootstrap-failing-o...

[#3]

Thanks for your reply!

I'm using the latest Cloudera Director and Manager 2.0.0 and 5.5.1

We have the following settings in the config which looks correct for me but it never worked with the preloaded AMI

...

repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/"
repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera"

...

products {
CDH: 5.5.1
}
parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.5.1/"]

...

[#4]

Still a open issue. Any suggestion would be really appreciated.

[#5]

Hi WZ,

I suspect that the parcel version on the AMI is different than what is being provided in the setup configuration. To help diagnose this further,

a) When you set up the AMI with the packer script, did you specify a specific version of CDH to use, or did it use the default? Could you provide the specific repository url that was used?

b) If you create an instance with the AMI, and ssh into it, what version of CDH parcel does it show? The paths to check would be /opt/cloudera/parcel-cache or /opt/cloudera/parcels

Let me know what you find, we can take it from there...

Thanks,

Jayita

[#6]

thanks a lot for your respons!

a. we are using the following parcel url

"parcel_url": "https://archive.cloudera.com/cdh5/parcels/5.5.1/CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel",

b. just lauched the instance using the baked AMI, and here are those dir showing. (note that parcels are empty because PREEXTRACT_PARCEL flag are set to false)

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcel-cache/

total 1453120

-rw-r--r--. 2 root root 1487987897 Mar  6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcel-repo/

total 1453124

-rw-r--r--. 2 root root 1487987897 Mar  6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

-rw-r--r--. 1 root root         41 Mar  6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel.sha

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcels

total 0

Thanks!

[#7]

Thanks for providing the information. I am trying to reproduce this and will get back to you.

[#8]

 Hi again,

I went through this setup, but was not able to reproduce the problem.

There may have been a few things different in what each of us tried. To diagnose further,

- did you run packer directly, or using build-ami.sh?

- did you override the cm url as well, or just cdh?

- was your ami redhat6 or some other variant / operating system? (the parcel installed is redhat6, but if it were redhat7 it would need a different one)

In my case I ran with build-ami.sh, using redhat6, and only overrode the cdh url, so I ended up with cm5.6 (it picks up the latest cm by default). My command was as follows, if you would like to try it:

PACKER_VARS="-var vpc_id=vpc-123456 -var subnet_id=subnet-123456 -var security_group_id=sg-123456 -var root_device_name=/dev/sda1"  sh build-ami.sh us-west-1 ami-6283a827 "my_ami_name" http://archive-primary.cloudera.com/cdh5/parcels/5.5.1/

My directory listing of /opt/cloudera/parcel-cache and /opt/cloudera/parcel-repo matches yours, so we know things worked until there...

A couple other places to check on your system:

•  Could you pull up the Cloudera Manager log to see if there are any errors around finding the parcel? You can find it on the instance at /var/log/cloudera-scm-server/cloudera-scm-server.log
  • In mine, after it starts Jetty server, I see this line "Discovered parcel on CM server: CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel". Does your log have any errors around finding the parcel?
• In the CM UI, top nav, there is an icon for parcels.
  • Under this, do you see any errors?
  • On this page if you click on Edit Settings, does the local parcel repository path show /opt/cloudera/parcel-repo and remote path show http://archive.cloudera.com/cdh5/parcels/5.5.1/ ?
• If you check the CM API for parcels, what does it return? http://<cm>:7180/api/v11/clusters/<clustername>/parcels

Thanks,

Jayita

[#9]

Thank you very much for the the response Jayita!

I will definitely give it another try in the morning. But we have a couple of questions regarding the cluster config for director:

0. is there any particular settings have to be set in order to work with the pre-loeaded ami?

1. in the instance settings, do we have to turn normalizeInstance off?

normalizeInstance: false

2. what if we need to setup additional parcels, do we need to do it in the pre-loaded scripts as well?

products {
CDH: 5.5.1
CLABS_PHOENIX: 4.5.2
KAFKA: 2.0.0
}
parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.5.1/",
"http://archive.cloudera.com/cloudera-labs/phoenix/parcels/1.2/",
"http://archive-primary.cloudera.com/kafka/parcels/2.0.0/"]

Thanks,

WZ

[#10]

Still seeing "java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels. ..."

Replied inline and thank you again for diagnose this with me!

A couple other places to check on your system:

•  Could you pull up the Cloudera Manager log to see if there are any errors around finding the parcel? 
•  In mine, after it starts Jetty server, I see this line "Discovered parcel on CM server: CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel". Does your log have any errors around finding the parcel?

I see the following in the log but there's no "Discovered parcel on CM server..." as yours:

2016-03-14 03:08:28,698 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel under /opt/cloudera/parcel-repo

and I found the following error in the log. Are those related?

2016-03-14 03:08:27,311 WARN MainThread:org.springframework.web.client.RestTemplate: GET request for "http://169.254.169.254/latest/meta-data/public-hostname" resulted in 404 (Not Found); invoking error handler

2016-03-14 03:08:27,312 ERROR MainThread:com.cloudera.server.web.cmf.cloud.EC2MetadataFetcher: Request to EC2 metadata failed: 404 Not Found

...

2016-03-14 03:07:08,536 ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: ERROR: relation "cm_version" does not exist

  Position: 21

2016-03-14 03:07:08,538 WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version, assuming no upgrade: org.hibernate.exception.SQLGrammarException: could not extract ResultSet

• In the CM UI, top nav, there is an icon for parcels.
  • Under this, do you see any errors?

I don't see any errors but only:

CDH 5

5.5.1-1.cdh5.5.1.p0.11

Downloaded

• On this page if you click on Edit Settings, does the local parcel repository path show /opt/cloudera/parcel-repo and remote path show http://archive.cloudera.com/cdh5/parcels/5.5.1/ ?

Yes, those two settings are correct.

• If you check the CM API for parcels, what does it return? http://<cm>:7180/api/v11/clusters/<clustername>/parcels

Here are the response from http://10.0.1.242:7180/api/v11/clusters/platform/parcels

{

• items:[

  • {

    • product: "CDH",
    • version: "5.5.1-1.cdh5.5.1.p0.11",
    • stage: "DOWNLOADED",
    • clusterRef:{

      • clusterName: "platform"

      }

    }

  ]

}