Member since
09-12-2016
21
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1709 | 10-09-2016 05:42 AM | |
| 2167 | 09-23-2016 08:18 AM |
11-16-2016
08:58 AM
Recently deployed CDH 5.8 on AWS using Director. Two versions of the jdk are downloaded [jdk1.6.0_31, jdk1.7.0_67-cloudera], unfortunately the symbolic link points to the outdated version. I say outdated as I think it is HIVE or HDFS that has been compiled with the more recent version. While it is easy to change the Java Home or the symbolic link "latest" to the correct version, how come the deployment does not perform this activity?
... View more
- Tags:
- Java Home
Labels:
10-09-2016
05:42 AM
All, We can set the parameters if we do the following and it was a tough lesson to learn.. There is a config session for Service wide parameters and then for specific areas, do not list or mix which parameters exist for which session as it will throw an error, but not an intelligent one. Additionally, I would recommend before setting parameters to run the script to dump the entire configuration into a file and then to review the specific parameters and the naming of them. I was using the GUI and seeing items like HBASE.replication and thought that this could be used, which was not the case. Instead, it was HBASE_enable_replication. I also found what I believe was a naming issue in that the name of the parameter did not align with the value, it wasn't spelled correctly like "..ion" and was "..on". The problem was I was spelling according to english vs reviewing the parameter name.
... View more
10-04-2016
01:53 PM
thank you for the timely follow-up... still trying to figure our configs section though ;-(
... View more
10-04-2016
01:13 PM
From first glance, when running the bootstrap command it appears as though you can run in multiple times and that logic might exist for it take take in any changes and incorporate them. This is based on my personal observation of running it, as it prompts the user informing that an existing configuration already exists and do they want to rebuild or resume. I have been trying to debug configuring the "Configs" section, both at the System-wide and at the Role-level. Given that the script errors out, i presumed that I could edit or update and then re-run the file. My oberservation is that the re-run does not take into account any changes made to the file being submitted, as i ended up commenting out a whole section of configs and the error continued to show-up. I then terminated and re-ran the script and everything worked fine (aka, I had no parameters in the config section). So, beware...
... View more
10-04-2016
08:30 AM
fyi - there is a type in the regionserver, as it only requires a single s
... View more
10-04-2016
08:29 AM
Problem not resolved but now get the following error [2016-10-04 07:19:05] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Applying custom configurations of services [2016-10-04 07:19:05] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=ServicesResource.updateServiceConfig",causeClass=class javax.ws.rs.BadRequestException, causeMessage="HTTP 400 Bad Request"} at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:321) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:100) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2] at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_101] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_101] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$94a41497.runUnchecked(<generated>) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na] at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na] at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.1.0.jar!/:2.1.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=ServicesResource.updateServiceConfig at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.1.0.jar!/:2.1.0] at com.sun.proxy.$Proxy182.updateServiceConfig(Unknown Source) ~[na:na] at com.cloudera.launchpad.bootstrap.cluster.AddServices.customConfigureServices(AddServices.java:504) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.cluster.AddServices.customConfigureServices(AddServices.java:466) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:304) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] ... 33 common frames omitted [2016-10-04 07:19:05] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
... View more
10-03-2016
03:07 PM
I tried changing this to a Roletype of Master and it didn't work I think I found the problem and am moving this to the Cluster definition. I believe the error may be how I am setting the values and think they should be w/o the double quotes..
... View more
10-03-2016
12:36 PM
All, I have successfully configured an AWS cluster with 8 nodes up and running. I have posted questions on how to confgure services, which I understand can be done via the API but was seeking to do this via the configuration file that is shipped to CD to create the CM and the rest of the nodes. However, when I set the values I do not see the any change in the values (using the CM UI). Below is my example.. when I try to set the HBASE parameters I do the following... roles { HDFS: [ JOURNALNODE, BALANCER] ZOOKEEPER: [SERVER] HBASE: [MASTER] } # # Configure services per ML team # config { HBASE { REGIONSERVER { hbase_regionsserver_msginterval: "25" hbase_hstore_compactionThreshold: "15" hbase_hstore_blockStoreFiles: "35" hbase_hstore_blockingWaitTime: "10" hbase_enable_replication: "true" hbase_snapshot_enabled: "true" hbase_enable_indexing: "true" hbase_active_master_detection_window: "10" hbase_server_dfs_client_hedged_read_threadpool_size: "10" hbase_regionserver_regionmover_thread_count: "4" } } and I don't see a change in the values. I patterned this after how Cloudera shows how to do the HA Config, but can't seem to get this to work.. ideas?
... View more
10-02-2016
05:35 AM
Curious.. can't we set the values for Hbase, HDFS, etc using the configuration file with the "Configs {} section after we identify the services?
... View more
09-30-2016
02:51 PM
I tore down the cluster already. In the configuration file, I request something like the following... cluster { products { CDH: 5 configs { # HDFS fencing should be set to true for HA configurations HDFS { dfs_ha_fencing_methods: "shell(true)" } } services: [HDFS, ZOOKEEPER, HBASE ] I am trying to load unique settings for HBASE I found them in the API and they are API Name hbase_regionserver_msginterval hbase_hstore_compactionThreshold hbase_hstore_blockStoreFiles hbase_hstore_blockingWaitTime hbase_enable_replication hbase_snapshot_enabled hbase_enable_indexing hbase_active_master_detection_window hbase_server_dfs_client_hdeged_read_threadpool_size hbase_regionserver_regionmover_thread_count My understanding is that these need to be done in the post creation set-up with the CM API or I am thinking that I could set them very similar to how HA is done above with setting the HDFS parameter. If it is via the CM API, I want to know the name w/o looking at a UI.
... View more
09-30-2016
02:12 PM
1 Kudo
After studying the CM API and standing up and instance, I believe I have a handle on configuring settings for given parameters. Our direction is to fully-script the install w/o any intervention thereby making the implementation idempotent. However, upon reviewing the client configuration file .. I see that I can name the cluster, but the actual name of the service appears to be something random and this makes it very difficult to program how to capture the name for the service. Example: HDFS "name" : "CD-HDFS-vbggAOii",
... View more
09-29-2016
02:48 PM
Having configured a base system on AWS with CM, I want to programtically update the environment - hoefully through the use of the configuration file, to update specific paramaters. I found the API relevant names, example, hbase_regionserver_msginterval and want to change this from the default to 25. How is this to be done?
... View more
09-23-2016
08:18 AM
Below is my script that I got running.. but sanitized AWS credentials, network, security.. etc In shows how to attach multiple EBS volumes (2), set delete on terminate so that they clean up, tag the volumes/resources ################################################################################ # begin # name: INSERT-NAME-OF-CLUSTER #Ultimately should have comments on each line for understanding. provider { type: aws region: us-west-2 subnetId: INSERT AWS Subnet ID securityGroupsIds: INSERT SG-ID or create IAM Profile instanceNamePrefix: INSERT-NAME-OF-CLUSTER rootVolumeSizeGB: 100 rootVolumeType: gp2 # associatePublicIpAddresses: true # spotBimi-e3d81583dUSDPerHr: 0.50 } ssh { username: ec2-user # for RHEL image privateKey: keypair.pem # with an absolute path to .pem file } instances { m42x { type: m4.2xlarge # requires an HVM AMI image: INSERT AMI rootVolumeSizeGB: 100 # iamProfileName: iam-profile-REPLACE-ME tags { owner: ${?USER} Project: "INSERT Project name" Vertical: "Insert Project vertical" } bootstrapScript: """#!/bin/sh # # This is an embedded bootstrap script that runs as root and can be used to customize # the instances immediately after boot and before any other Cloudera Director action export AWS_ACCESS_KEY_ID=INSERT AWS Access Key export AWS_SECRET_ACCESS_KEY=INSERT AWS Secret Key export INSTANCE_ID=$(curl http://instance-data/latest/meta-data/instance-id) export AVAILABILITY_ZONE=$(curl http://instance-data/latest/meta-data/placement/availability-zone) export VOL1_ID=$(aws ec2 create-volume --size 15 --region us-west-2 --availability-zone ${AVAILABILITY_ZONE} --volume-type gp2 --query "VolumeId" | tr -d '"') export VOL2_ID=$(aws ec2 create-volume --size 15 --region us-west-2 --availability-zone ${AVAILABILITY_ZONE} --volume-type gp2 --query "VolumeId" | tr -d '"') echo 'Sleeping 30' sleep 30 echo 'Create additional EBS volumes' aws ec2 attach-volume --region us-west-2 --volume-id ${VOL1_ID} --instance-id ${INSTANCE_ID} --device /dev/sdf aws ec2 attach-volume --region us-west-2 --volume-id ${VOL2_ID} --instance-id ${INSTANCE_ID} --device /dev/sdg # Create filesystems on the devices echo 'Create filesystems' mkfs.ext4 /dev/xvdf mkfs.ext4 /dev/xvdg # Create directories for the mount points mkdir /data1 mkdir /data2 # Add the mount points to /etc/fstab echo "/dev/xvdf /data1 ext4 defaults 0 0" >> /etc/fstab echo "/dev/xvdg /data2 ext4 defaults 0 0" >> /etc/fstab sleep 10 # Mount all the devices echo 'Mount drives' mount -a # # modify attributes of new volumes to delete on terminate echo 'Modify volume attributes' aws ec2 modify-instance-attribute --instance-id ${INSTANCE_ID} --region us-west-2 --block-device-mappings "[{\"DeviceName\": \"/dev/sdf\",\"Ebs\":{\"DeleteOnTermination\":true}}]" aws ec2 modify-instance-attribute --instance-id ${INSTANCE_ID} --region us-west-2 --block-device-mappings "[{\"DeviceName\": \"/dev/sdg\",\"Ebs\":{\"DeleteOnTermination\":true}}]" # # tag the volumes echo 'Tag resources' aws ec2 create-tags --region us-west-2 --resources ${INSTANCE_ID} --tags Key=Application,Value=BIGDATA Key=Vertical,Value=VerticalName Key=Project,Value="Project name" aws ec2 create-tags --region us-west-2 --resources ${VOL1_ID} --tags Key=Application,Value=BIGDATA Key=Vertical,Value=VerticalName Key=Project,Value="Project name" aws ec2 create-tags --region us-west-2 --resources ${VOL2_ID} --tags Key=Application,Value=BIGDATA Key=Vertical,Value=VerticalName Key=Project,Value="Project name" # # If the exit code is not zero Cloudera Director will automatically retry echo 'This is the last line of the script' exit 0 """ # # Flag indicating whether to normalize the instance. Not setting normalization here implies that your # bootstrap script will take care of normalization. This is an advanced configuration that will require # assistance from Cloudera support. # # Normalization includes: # downloading and installing packages # minimizing swappiness # increasing the maximun number of open files # mounting ephemeral disks # resizing the root partition. # # Defaults to true # normalizeInstance: true } m44x { type: m4.4xlarge image: ami tags { owner: ${?USER} } } c34x { type: c3.4xlarge image: ami tags { owner: ${?USER} } } i2x { type: i2.xlarge image: ami tags { owner: ${?USER} } } i22x { type: i2.2xlarge image: ami tags { owner: ${?USER} } } r32x { type: r3.2xlarge image: ami tags { owner: ${?USER} } } t2l { # only suitable as a gateway type: t2.large image: ami tags { owner: ${?USER} } } } cloudera-manager { instance: ${instances.m42x} { tags { # add any additional tags as needed Application: "Cloudera Manager 5" Project: "place project name here" Vertical: "place vertical name here" } } repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.8/" repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera" enableEnterpriseTrial: true } cluster { products { CDH: 5 } configs { # HDFS fencing should be set to true for HA configurations HDFS { dfs_ha_fencing_methods: "shell(true)" } } services: [HDFS, ZOOKEEPER, HBASE ] hdfsmaster-1 { count: 2 instance: ${instances.m44x} { tags { group: hdfsmaster-1 } } roles { HDFS: [NAMENODE, JOURNALNODE, FAILOVERCONTROLLER ] ZOOKEEPER: [SERVER] } # NameNode nameservice, autofailover, and quorum journal name must be configured for high availability configs { HDFS { NAMENODE { dfs_federation_namenode_nameservice: hanameservice autofailover_enabled: true dfs_namenode_quorum_journal_name: hanameservice } } } } hdfsmaster-2 { count: 1 instance: ${instances.m44x} { tags { group: hdfsmaster-2 } } roles { HDFS: [ JOURNALNODE, BALANCER] ZOOKEEPER: [SERVER] HBASE: [MASTER] } } master-1 { count: 1 instance: ${instances.m44x} { tags { group: master } } roles { HBASE: [MASTER] ZOOKEEPER: [SERVER] } } workers { count: 3 minCount: 1 instance: ${instances.m42x} { # placementGroup: REPLACE-ME tags { group: worker } } roles { HDFS: [DATANODE] HBASE: [REGIONSERVER] } } postCreateScripts: ["""#!/bin/sh echo 'Hello World!' exit 0 """, """#!/usr/bin/python print 'Hello again!' """] preTerminateScripts: ["""#!/bin/sh echo 'Goodbye World!' exit 0 """, """#!/usr/bin/python print 'Goodbye again!' """] }
... View more
09-21-2016
04:15 PM
I can successfully create a cluster using the Cloudera provided template and nodes following the AWS template profile. Am seeking to script how to configure parameters such as hbase.regionserver.msginterval hbase.hstore.blockingStoreFiles hbase.snapshot.enabled... I presume that these will go in the post creation script... how?
... View more
09-16-2016
02:41 PM
Using the following Configuration file, it dies... This is patterned off of the provided example from Cloudera.. it builds fine when hdfs-master count = 1, but when making it 2 to have more than a single NameNode, etc.. it fails.. cloudera-manager { instance: ${instances.m42x} { tags { # add any additional tags as needed application: "Cloudera Manager 5" } } repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.8/" repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera" enableEnterpriseTrial: true } cluster { products { CDH: 5 } configs { # HDFS fencing should be set to true for HA configurations HDFS { dfs_ha_fencing_methods: "shell(true)" } } services: [HDFS, ZOOKEEPER, HBASE, HIVE ] hdfsmaster-1 { count: 2 instance: ${instances.m42x} { tags { group: hdfsmaster-1 } } roles { HDFS: [NAMENODE, JOURNALNODE, FAILOVERCONTROLLER, BALANCER] ZOOKEEPER: [SERVER] HBASE: [MASTER] } # NameNode nameservice, autofailover, and quorum journal name must be configured for high availability configs { HDFS { NAMENODE { dfs_federation_namenode_nameservice: hanameservice autofailover_enabled: true dfs_namenode_quorum_journal_name: hanameservice } } } } hdfsmaster-2 { count: 1 instance: ${instances.m42x} { tags { group: hdfsmaster-2 } } roles { HDFS: [ JOURNALNODE] ZOOKEEPER: [SERVER] } } master-1 { count: 1 instance: ${instances.m42x} { tags { group: master } } roles { HBASE: [MASTER] ZOOKEEPER: [SERVER] } } workers { count: 3 minCount: 1 instance: ${instances.c34x} { # placementGroup: REPLACE-ME tags { group: worker } } roles { HDFS: [DATANODE] HBASE: [REGIONSERVER] } } gateways { count: 1 instance: ${instances.t2l} { tags { group: gateway } } roles { HIVE: [GATEWAY] } } postCreateScripts: ["""#!/bin/sh echo 'Hello World!' exit 0 """, """#!/usr/bin/python print 'Hello again!' """] preTerminateScripts: ["""#!/bin/sh echo 'Goodbye World!' exit 0 """, """#!/usr/bin/python print 'Goodbye again!' """]
... View more
09-15-2016
10:56 AM
Doing a fresh build in AWS and the CM (2.1) reports... Sep 15, 1:35:18.088 PM ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper ERROR: relation "cm_version" does not exist
... View more
09-13-2016
08:21 AM
After a bit of futher checking (and I apologize for posting to the wrong group) There was not enough disk space being allocated ... this is the only item I changed and then the deployment worked. I think this goes to show that the details on reporting needs addressing.
... View more
09-13-2016
08:19 AM
thanks.. my bad
... View more
09-12-2016
12:05 PM
[2016-09-12 10:16:59] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [67, Environment{name='ja-test-cluster Environment', provider=InstanceProviderConfig{type='aws'}, cr ... [2016-09-12 10:16:59] INFO [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=67, name=CreateHiveDatabase, startTime=Mon Sep 12 10:16:59 GMT-08:00 2016, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=ja-test-cluster, serviceName=CD-HIVE-kYKDwqno}, roleRef=null, hostRef=null, parent=null} [2016-09-12 10:17:14] INFO [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateHiveDatabase with ID 67 completed successfully. Details: ApiCommand{id=67, name=CreateHiveDatabase, startTime=Mon Sep 12 10:16:59 GMT-08:00 2016, endTime=Mon Sep 12 10:17:04 GMT-08:00 2016, active=false, success=true, resultMessage=Created Hive Metastore Database., serviceRef=ApiServiceRef{peerName=null, clusterName=ja-test-cluster, serviceName=CD-HIVE-kYKDwqno}, roleRef=null, hostRef=null, parent=null} [2016-09-12 10:17:14] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << None{} [2016-09-12 10:17:14] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [68, Environment{name='ja-test-cluster Environment', provider=InstanceProviderConfig{type='aws'}, cr ... [2016-09-12 10:17:14] INFO [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateSentryDatabase with ID 68 completed successfully. Details: ApiCommand{id=68, name=CreateSentryDatabase, startTime=Mon Sep 12 10:16:59 GMT-08:00 2016, endTime=Mon Sep 12 10:17:04 GMT-08:00 2016, active=false, success=true, resultMessage=Successfully created Sentry database., serviceRef=ApiServiceRef{peerName=null, clusterName=ja-test-cluster, serviceName=CD-SENTRY-ctUWighV}, roleRef=null, hostRef=null, parent=null} [2016-09-12 10:17:14] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << None{} [2016-09-12 10:17:15] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> InvokeFirstRunClusterCommandV7/3 [Environment{name='ja-test-cluster Environment', provider=InstanceProviderConfig{type='aws'}, creden ... [2016-09-12 10:17:15] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Calling firstRun on cluster ja-test-cluster [2016-09-12 10:17:16] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << None{} [2016-09-12 10:17:16] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for firstRun on cluster ja-test-cluster] [2016-09-12 10:17:16] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Waiting for firstRun on cluster ja-test-cluster [2016-09-12 10:17:16] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << None{} [2016-09-12 10:17:16] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [69, Environment{name='ja-test-cluster Environment', provider=InstanceProviderConfig{type='aws'}, cr ... [2016-09-12 10:17:16] INFO [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=69, name=First Run, startTime=Mon Sep 12 10:17:15 GMT-08:00 2016, endTime=null, active=true, success=null, resultMessage=null, serviceRef=null, roleRef=null, hostRef=null, parent=null} [2016-09-12 10:17:31] INFO [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Collecting and downloading diagnostic data [2016-09-12 10:17:31] INFO [pipeline-thread-1] - c.c.l.b.ClouderaManagerLogRetriever: Diagnostic data collection initiated, command ID 76 [2016-09-12 10:18:26] INFO [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Attempting download of diagnostic data [2016-09-12 10:18:26] WARN [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Failed to download diagnostic data java.net.UnknownHostException: ip-10-231-17-119.us-west-2.compute.internal: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.7.0_101] at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:922) ~[na:1.7.0_101] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1316) ~[na:1.7.0_101] at java.net.InetAddress.getAllByName0(InetAddress.java:1269) ~[na:1.7.0_101] at java.net.InetAddress.getAllByName(InetAddress.java:1185) ~[na:1.7.0_101] at java.net.InetAddress.getAllByName(InetAddress.java:1119) ~[na:1.7.0_101] at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.2.jar!/:4.2] at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:278) ~[httpclient-4.2.jar!/:4.2] at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:162) ~[httpclient-4.2.jar!/:4.2] at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) ~[httpclient-4.2.jar!/:4.2] at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:641) ~[httpclient-4.2.jar!/:4.2] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) ~[httpclient-4.2.jar!/:4.2] at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ~[httpclient-4.2.jar!/:4.2] at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) ~[httpclient-4.2.jar!/:4.2] at com.cloudera.launchpad.common.http.HttpDownloader.download(HttpDownloader.java:188) ~[launchpad-common-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.common.http.HttpDownloader.download(HttpDownloader.java:147) ~[launchpad-common-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.collectAndDownloadDiagnosticData(UnboundedWaitForApiCommand.java:135) [launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:81) [launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:42) [launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.codahale.metrics.Timer.time(Timer.java:101) [metrics-core-3.1.2.jar!/:3.1.2] at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_101] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_101] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$66a4c1b5.runUnchecked(<generated>) [launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na] at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na] at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.1.0.jar!/:2.1.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] [2016-09-12 10:18:26] INFO [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Diagnostic data available at http://ip-10-231-17-119.us-west-2.compute.internal:7180/cmf/command/76/download [2016-09-12 10:18:26] ERROR [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Command First Run with ID 69 failed. Details: ApiCommand{id=69, name=First Run, startTime=Mon Sep 12 10:17:15 GMT-08:00 2016, endTime=Mon Sep 12 10:17:30 GMT-08:00 2016, active=false, success=false, resultMessage=Failed to perform First Run of services., serviceRef=null, roleRef=null, hostRef=null, parent=null} [2016-09-12 10:18:26] ERROR [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Command DeployClusterClientConfig with ID 70 failed. Details: ApiCommand{id=70, name=DeployClusterClientConfig, startTime=Mon Sep 12 10:17:15 GMT-08:00 2016, endTime=Mon Sep 12 10:17:30 GMT-08:00 2016, active=false, success=false, resultMessage=Failed to deploy client configuration to the cluster., serviceRef=null, roleRef=null, hostRef=null, parent=null} [2016-09-12 10:18:26] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager 'First Run' command execution failed: Failed to perform First Run of services. at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:89) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:42) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2] at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_101] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_101] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$66a4c1b5.runUnchecked(<generated>) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na] at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na] at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.1.0.jar!/:2.1.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] [2016-09-12 10:18:26] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager 'First Run' command execution failed: Failed to perform First Run of services. at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:89) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:42) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2] at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_101] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_101] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE] at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$66a4c1b5.runUnchecked(<generated>) ~[launchpad-bootstrap-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na] at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na] at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.1.0.jar!/:2.1.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] [2016-09-12 10:18:26] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Pipeline '23a0a994-ba95-4f09-b7ab-daa3243380e8' failed at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$66a4c1b5 at com.cloudera.launchpad.bootstrap.cluster.firstrun.InvokeFirstRunClusterCommandV7:1 [2016-09-12 10:18:26] INFO [pipeline-thread-1] - c.c.l.p.s.PipelineRepositoryService: Pipeline '23a0a994-ba95-4f09-b7ab-daa3243380e8': RUNNING -> ERROR [2016-09-12 10:18:29] INFO [pipeline-thread-1] - c.c.l.d.ClusterRepositoryService: Cluster 'ja-test-cluster': BOOTSTRAPPING -> BOOTSTRAP_FAILED [2016-09-12 10:18:33] INFO [Thread-9] - o.s.c.a.AnnotationConfigApplicationContext: Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@f76eb40: startup date [Mon Sep 12 09:26:03 GMT-08:00 2016]; root of context hierarchy [2016-09-12 10:18:34] INFO [Thread-9] - o.s.o.j.LocalContainerEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit 'default' [2016-09-12 10:18:34] WARN [Thread-9] - c.c.l.c.security.CipherSchemeFactory: Allowing cipher scheme to be set again, prior scheme is com.cloudera.launchpad.common.security.TripleDESCipher@da9df77 ec2-user@c1-r-3mls-cdjrf01:~/.cloudera-director/logs$
... View more
