Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issue with AMI with Parcel Preloaded

Solved Go to solution

Issue with AMI with Parcel Preloaded

Explorer

I built an AMI using the preloaded-ami-builder script: 

https://github.com/cloudera/director-scripts/tree/master/preloaded-ami-builder

and my director config have the following specifying the parcel version I put in the AMI.:

 

products {
CDH: 5.5.1
}
parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.5.1/"]

 

But I keep getting this "CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels" exception when bootstrap-remote the config. Everything works if I remove the parcel_provisioner.sh in the packer though it takes time downloading the parcel. Please help me. I've spent days searching for solutions but no luck yet.

 

Another question not related to this issue but I see the builder has steps installing cloudera-manager-server and  cloudera-manager-daemons, why I am still seeing the following in the director provisioning step?

Installing cloudera-manager-server package

Installing cloudera-manager-daemons package

 

...

I suppose those should be skipped as well but please help me understand this.

 

 

Thanks!

 

-----------------------------------------------------------------------------------------------------------------------------------------------------

 

2016-01-19 17:50:03] INFO [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: >> EnableParcels/3 [CreateClusterContext{environment=Environment{name='platform Environment', provider=InstanceProvider ...
[2016-01-19 17:50:04] ERROR [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:77) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:36) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_65]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels$$EnhancerBySpringCGLIB$$7a951cb.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) [launchpad-pipeline-database-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-1.5.2.jar!/:1.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.util.BootstrapClusterUtils.getParcelProducts(BootstrapClusterUtils.java:319) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:57) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
... 33 common frames omitted

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Issue with AMI with Parcel Preloaded

Explorer

Thanks for your help! It finally works now!

 

Turns out that "normalizeInstance" has to set to true. It's a bit confusing though as in the comment it says the following and I thought most of them are already done in the preloaded AMI.

# Normalization includes:
# downloading and installing packages
# minimizing swappiness
# increasing the maximun number of open files
# mounting ephemeral disks
# resizing the root partition.

I will modify the packer scripts to download additional parcels we need. It will be great if you add support semi-preloeaded AMI so small parcels can be downloaded during provisioning.

11 REPLIES 11

Re: Issue with AMI with Parcel Preloaded

Expert Contributor

What versions of Cloudera Director and Cloudera Manager are you using?

 

You may want to take a look at this comment for possible info on problems with version incompatibilities: http://community.cloudera.com/t5/Cloudera-Director-Cloud-based/Cloudera-Director-bootstrap-failing-o...

Highlighted

Re: Issue with AMI with Parcel Preloaded

Explorer

Thanks for your reply!

I'm using the latest Cloudera Director and Manager 2.0.0 and 5.5.1

 

We have the following settings in the config which looks correct for me but it never worked with the preloaded AMI

...

repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/"
repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera"

...

products {
CDH: 5.5.1
}
parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.5.1/"]

...

Re: Issue with AMI with Parcel Preloaded

Explorer

Still a open issue. Any suggestion would be really appreciated.

Re: Issue with AMI with Parcel Preloaded

Contributor

Hi WZ,

 

I suspect that the parcel version on the AMI is different than what is being provided in the setup configuration. To help diagnose this further,

 

a) When you set up the AMI with the packer script, did you specify a specific version of CDH to use, or did it use the default? Could you provide the specific repository url that was used?

 

b) If you create an instance with the AMI, and ssh into it, what version of CDH parcel does it show? The paths to check would be /opt/cloudera/parcel-cache or /opt/cloudera/parcels

 

Let me know what you find, we can take it from there...

 

Thanks,

Jayita

Re: Issue with AMI with Parcel Preloaded

Explorer

thanks a lot for your respons!

 

a. we are using the following parcel url

"parcel_url": "https://archive.cloudera.com/cdh5/parcels/5.5.1/CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel",

b. just lauched the instance using the baked AMI, and here are those dir showing. (note that parcels are empty because PREEXTRACT_PARCEL flag are set to false)

 

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcel-cache/

total 1453120

-rw-r--r--. 2 root root 1487987897 Mar  6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

 

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcel-repo/

total 1453124

-rw-r--r--. 2 root root 1487987897 Mar  6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

-rw-r--r--. 1 root root         41 Mar  6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel.sha

 

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcels

total 0

 

 

Thanks!

Re: Issue with AMI with Parcel Preloaded

Contributor

Thanks for providing the information. I am trying to reproduce this and will get back to you.

Re: Issue with AMI with Parcel Preloaded

Contributor

 Hi again,

 

I went through this setup, but was not able to reproduce the problem.

 

There may have been a few things different in what each of us tried. To diagnose further,

- did you run packer directly, or using build-ami.sh?

- did you override the cm url as well, or just cdh?

- was your ami redhat6 or some other variant / operating system? (the parcel installed is redhat6, but if it were redhat7 it would need a different one)

 

In my case I ran with build-ami.sh, using redhat6, and only overrode the cdh url, so I ended up with cm5.6 (it picks up the latest cm by default). My command was as follows, if you would like to try it:

PACKER_VARS="-var vpc_id=vpc-123456 -var subnet_id=subnet-123456 -var security_group_id=sg-123456 -var root_device_name=/dev/sda1"  sh build-ami.sh us-west-1 ami-6283a827 "my_ami_name" http://archive-primary.cloudera.com/cdh5/parcels/5.5.1/

 

My directory listing of /opt/cloudera/parcel-cache and /opt/cloudera/parcel-repo matches yours, so we know things worked until there...

 

A couple other places to check on your system:

  •  Could you pull up the Cloudera Manager log to see if there are any errors around finding the parcel? You can find it on the instance at /var/log/cloudera-scm-server/cloudera-scm-server.log
    • In mine, after it starts Jetty server, I see this line "Discovered parcel on CM server: CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel". Does your log have any errors around finding the parcel?
  • In the CM UI, top nav, there is an icon for parcels.
  • If you check the CM API for parcels, what does it return? http://<cm>:7180/api/v11/clusters/<clustername>/parcels

 

Thanks,

Jayita

 

Re: Issue with AMI with Parcel Preloaded

Explorer

Thank you very much for the the response Jayita!

I will definitely give it another try in the morning. But we have a couple of questions regarding the cluster config for director:

0. is there any particular settings have to be set in order to work with the pre-loeaded ami?

1. in the instance settings, do we have to turn normalizeInstance off?

normalizeInstance: false

2. what if we need to setup additional parcels, do we need to do it in the pre-loaded scripts as well?

products {
CDH: 5.5.1
CLABS_PHOENIX: 4.5.2
KAFKA: 2.0.0
}
parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.5.1/",
"http://archive.cloudera.com/cloudera-labs/phoenix/parcels/1.2/",
"http://archive-primary.cloudera.com/kafka/parcels/2.0.0/"]

 

 

 

Thanks,

 

WZ

Re: Issue with AMI with Parcel Preloaded

Explorer

Still seeing "java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels. ..."

Replied inline and thank you again for diagnose this with me!

 

A couple other places to check on your system:

  •  Could you pull up the Cloudera Manager log to see if there are any errors around finding the parcel? 
  •  In mine, after it starts Jetty server, I see this line "Discovered parcel on CM server: CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel". Does your log have any errors around finding the parcel?

I see the following in the log but there's no "Discovered parcel on CM server..." as yours:

2016-03-14 03:08:28,698 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel under /opt/cloudera/parcel-repo

and I found the following error in the log. Are those related?

2016-03-14 03:08:27,311 WARN MainThread:org.springframework.web.client.RestTemplate: GET request for "http://169.254.169.254/latest/meta-data/public-hostname" resulted in 404 (Not Found); invoking error handler

2016-03-14 03:08:27,312 ERROR MainThread:com.cloudera.server.web.cmf.cloud.EC2MetadataFetcher: Request to EC2 metadata failed: 404 Not Found

...

2016-03-14 03:07:08,536 ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: ERROR: relation "cm_version" does not exist

  Position: 21

2016-03-14 03:07:08,538 WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version, assuming no upgrade: org.hibernate.exception.SQLGrammarException: could not extract ResultSet

 

  • In the CM UI, top nav, there is an icon for parcels.
    • Under this, do you see any errors?

I don't see any errors but only:

CDH 55.5.1-1.cdh5.5.1.p0.11
Downloaded

Yes, those two settings are correct.

 

  • If you check the CM API for parcels, what does it return? http://<cm>:7180/api/v11/clusters/<clustername>/parcels

Here are the response from http://10.0.1.242:7180/api/v11/clusters/platform/parcels

{

  • items:
     
    [
    •  
      {
      • product: "CDH",
      • version: "5.5.1-1.cdh5.5.1.p0.11",
      • stage: "DOWNLOADED",
      • clusterRef:
         
        {
        • clusterName: "platform"
        }
      }
    ]

}

Don't have an account?
Coming from Hortonworks? Activate your account here