question Re: Issue with AMI with Parcel Preloaded in Support Questions

Issue with AMI with Parcel Preloaded

WZ — Fri, 16 Sep 2022 09:58:51 GMT

I built an AMI using the preloaded-ami-builder script:

https://github.com/cloudera/director-scripts/tree/master/preloaded-ami-builder

and my director config have the following specifying the parcel version I put in the AMI.:

products {
CDH: 5.5.1
}
parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.5.1/"]

But I keep getting this "CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels" exception when bootstrap-remote the config. Everything works if I remove the parcel_provisioner.sh in the packer though it takes time downloading the parcel. Please help me. I've spent days searching for solutions but no luck yet.

Another question not related to this issue but I see the builder has steps installing cloudera-manager-server and cloudera-manager-daemons, why I am still seeing the following in the director provisioning step?

Installing cloudera-manager-server package

Installing cloudera-manager-daemons package

...

I suppose those should be skipped as well but please help me understand this.

Thanks!

-----------------------------------------------------------------------------------------------------------------------------------------------------

2016-01-19 17:50:03] INFO [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: >> EnableParcels/3 [CreateClusterContext{environment=Environment{name='platform Environment', provider=InstanceProvider ...
[2016-01-19 17:50:04] ERROR [pipeline-thread-2] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:77) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:36) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_65]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_65]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels$$EnhancerBySpringCGLIB$$7a951cb.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) [launchpad-pipeline-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) [launchpad-pipeline-database-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-1.5.2.jar!/:1.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels.
at com.cloudera.launchpad.bootstrap.cluster.util.BootstrapClusterUtils.getParcelProducts(BootstrapClusterUtils.java:319) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
at com.cloudera.launchpad.bootstrap.cluster.EnableParcels.run(EnableParcels.java:57) ~[launchpad-bootstrap-1.5.2.jar!/:1.5.2]
... 33 common frames omitted

Re: Issue with AMI with Parcel Preloaded

jadair — Fri, 22 Jan 2016 19:49:58 GMT

What versions of Cloudera Director and Cloudera Manager are you using?

You may want to take a look at this comment for possible info on problems with version incompatibilities: http://community.cloudera.com/t5/Cloudera-Director-Cloud-based/Cloudera-Director-bootstrap-failing-on-Installing-cloudera/m-p/35949#M480

Re: Issue with AMI with Parcel Preloaded

WZ — Fri, 22 Jan 2016 20:14:24 GMT

Thanks for your reply!

I'm using the latest Cloudera Director and Manager 2.0.0 and 5.5.1

We have the following settings in the config which looks correct for me but it never worked with the preloaded AMI

...

repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/"
repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera"

...

products {
CDH: 5.5.1
}
parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.5.1/"]

...

Re: Issue with AMI with Parcel Preloaded

WZ — Tue, 08 Mar 2016 03:59:03 GMT

Still a open issue. Any suggestion would be really appreciated.

Re: Issue with AMI with Parcel Preloaded

Jayita — Tue, 08 Mar 2016 18:59:26 GMT

Hi WZ,

I suspect that the parcel version on the AMI is different than what is being provided in the setup configuration. To help diagnose this further,

a) When you set up the AMI with the packer script, did you specify a specific version of CDH to use, or did it use the default? Could you provide the specific repository url that was used?

b) If you create an instance with the AMI, and ssh into it, what version of CDH parcel does it show? The paths to check would be /opt/cloudera/parcel-cache or /opt/cloudera/parcels

Let me know what you find, we can take it from there...

Thanks,

Jayita

Re: Issue with AMI with Parcel Preloaded

WZ — Tue, 08 Mar 2016 19:42:18 GMT

thanks a lot for your respons!

a. we are using the following parcel url

"parcel_url": "https://archive.cloudera.com/cdh5/parcels/5.5.1/CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel",

b. just lauched the instance using the baked AMI, and here are those dir showing. (note that parcels are empty because PREEXTRACT_PARCEL flag are set to false)

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcel-cache/

total 1453120

-rw-r--r--. 2 root root 1487987897 Mar 6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcel-repo/

total 1453124

-rw-r--r--. 2 root root 1487987897 Mar 6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

-rw-r--r--. 1 root root 41 Mar 6 04:17 CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel.sha

[ec2-user@ip-10-0-1-254 ~]$ ll /opt/cloudera/parcels

total 0

Thanks!

Re: Issue with AMI with Parcel Preloaded

Jayita — Thu, 10 Mar 2016 18:16:32 GMT

Thanks for providing the information. I am trying to reproduce this and will get back to you.

Re: Issue with AMI with Parcel Preloaded

Jayita — Fri, 11 Mar 2016 21:48:27 GMT

Hi again,

I went through this setup, but was not able to reproduce the problem.

There may have been a few things different in what each of us tried. To diagnose further,

- did you run packer directly, or using build-ami.sh?

- did you override the cm url as well, or just cdh?

- was your ami redhat6 or some other variant / operating system? (the parcel installed is redhat6, but if it were redhat7 it would need a different one)

In my case I ran with build-ami.sh, using redhat6, and only overrode the cdh url, so I ended up with cm5.6 (it picks up the latest cm by default). My command was as follows, if you would like to try it:

PACKER_VARS="-var vpc_id=vpc-123456 -var subnet_id=subnet-123456 -var security_group_id=sg-123456 -var root_device_name=/dev/sda1" sh build-ami.sh us-west-1 ami-6283a827 "my_ami_name" http://archive-primary.cloudera.com/cdh5/parcels/5.5.1/

My directory listing of /opt/cloudera/parcel-cache and /opt/cloudera/parcel-repo matches yours, so we know things worked until there...

A couple other places to check on your system:

Could you pull up the Cloudera Manager log to see if there are any errors around finding the parcel? You can find it on the instance at /var/log/cloudera-scm-server/cloudera-scm-server.log
- In mine, after it starts Jetty server, I see this line "Discovered parcel on CM server: CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel". Does your log have any errors around finding the parcel?
In the CM UI, top nav, there is an icon for parcels.
- Under this, do you see any errors?
- On this page if you click on Edit Settings, does the local parcel repository path show /opt/cloudera/parcel-repo and remote path show http://archive.cloudera.com/cdh5/parcels/5.5.1/ ?
If you check the CM API for parcels, what does it return? http://<cm>:7180/api/v11/clusters/<clustername>/parcels

Thanks,

Jayita

Re: Issue with AMI with Parcel Preloaded

WZ — Mon, 14 Mar 2016 03:21:13 GMT

Thank you very much for the the response Jayita!

I will definitely give it another try in the morning. But we have a couple of questions regarding the cluster config for director:

0. is there any particular settings have to be set in order to work with the pre-loeaded ami?

1. in the instance settings, do we have to turn normalizeInstance off?

normalizeInstance: false

2. what if we need to setup additional parcels, do we need to do it in the pre-loaded scripts as well?

products {
CDH: 5.5.1
CLABS_PHOENIX: 4.5.2
KAFKA: 2.0.0
}
parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.5.1/",
"http://archive.cloudera.com/cloudera-labs/phoenix/parcels/1.2/",
"http://archive-primary.cloudera.com/kafka/parcels/2.0.0/"]

Thanks,

Re: Issue with AMI with Parcel Preloaded

WZ — Mon, 14 Mar 2016 07:32:57 GMT

Still seeing "java.lang.IllegalArgumentException: CDH=5.5.1-1.cdh5.5.1.p0.11 not found in list of all parcels. ..."

Replied inline and thank you again for diagnose this with me!

A couple other places to check on your system:

Could you pull up the Cloudera Manager log to see if there are any errors around finding the parcel?
In mine, after it starts Jetty server, I see this line "Discovered parcel on CM server: CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel". Does your log have any errors around finding the parcel?

I see the following in the log but there's no "Discovered parcel on CM server..." as yours:

2016-03-14 03:08:28,698 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel under /opt/cloudera/parcel-repo

and I found the following error in the log. Are those related?

2016-03-14 03:08:27,311 WARN MainThread:org.springframework.web.client.RestTemplate: GET request for "http://169.254.169.254/latest/meta-data/public-hostname" resulted in 404 (Not Found); invoking error handler

2016-03-14 03:08:27,312 ERROR MainThread:com.cloudera.server.web.cmf.cloud.EC2MetadataFetcher: Request to EC2 metadata failed: 404 Not Found

...

2016-03-14 03:07:08,536 ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: ERROR: relation "cm_version" does not exist

Position: 21

2016-03-14 03:07:08,538 WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version, assuming no upgrade: org.hibernate.exception.SQLGrammarException: could not extract ResultSet

In the CM UI, top nav, there is an icon for parcels.
- Under this, do you see any errors?

I don't see any errors but only:

CDH 5

5.5.1-1.cdh5.5.1.p0.11

Downloaded

On this page if you click on Edit Settings, does the local parcel repository path show /opt/cloudera/parcel-repo and remote path show http://archive.cloudera.com/cdh5/parcels/5.5.1/ ?

Yes, those two settings are correct.

If you check the CM API for parcels, what does it return? http://<cm>:7180/api/v11/clusters/<clustername>/parcels

Here are the response from http://10.0.1.242:7180/api/v11/clusters/platform/parcels

{

items:

[
- {
  - product: "CDH",
  - version: "5.5.1-1.cdh5.5.1.p0.11",
  - stage: "DOWNLOADED",
  - clusterRef:
    
    {
    clusterName: "platform"
    }
  }
]

}

Re: Issue with AMI with Parcel Preloaded

Mike Wilson — Mon, 14 Mar 2016 15:17:21 GMT

This is strange -- truthfully, it looks like all your settings are copacetic, yet Director is failing to detect them here. A few more questions:

1. Could you try http://10.0.1.242:7180/api/v7/clusters/platform/parcels (the URL difference is v7 vs. v11).
2. So, the parcel that's being downloaded now is specifically for RHEL/CentOS 6. What base AMI are you currently using to build preloaded AMIs for CM/your cluster? Is it possible there's a mismatch, i.e. using RHEL/CentOS 7? If this is so, you may have to modify build-ami.sh to look for the parcel ending in el7 instead of el6.

As for multiple parcels -- this is doable, but the scripts as written do not support this, unfortunately. At the moment, you can either do this manually, or you can modify the packer scripts to download these other parcels. The latter shouldn't be too bad, but would require some packer familiarity.

I'm going to file a ticket internally for this feature, however, as it seems like a useful one.

Re: Issue with AMI with Parcel Preloaded

WZ — Mon, 14 Mar 2016 18:27:24 GMT

Thanks for your help! It finally works now!

Turns out that "normalizeInstance" has to set to true. It's a bit confusing though as in the comment it says the following and I thought most of them are already done in the preloaded AMI.

# Normalization includes:
# downloading and installing packages
# minimizing swappiness
# increasing the maximun number of open files
# mounting ephemeral disks
# resizing the root partition.

I will modify the packer scripts to download additional parcels we need. It will be great if you add support semi-preloeaded AMI so small parcels can be downloaded during provisioning.