Member since
11-02-2017
15
Posts
0
Kudos Received
0
Solutions
10-05-2018
03:09 PM
Hi, After Deep dive investigation.. caught a Data parsing issue... This issue is Resolved...
... View more
10-04-2018
01:16 PM
Hi,
Cluster Version : Cloudera Express 5.14.3
No Recent changes since last 3 Months on the Cluster
Recently, a Week Back, I started getting the below Error and the Job Fails...Jobs are getting executed from Oozie.
Any Feedback ?
"ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask"
"Error: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=3)"
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.dynamic.partition.mode=nonstrict; No rows affected (0.005 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.max.dynamic.partitions=1000; No rows affected (0.004 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.max.dynamic.partitions.pernode=1000; No rows affected (0.004 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.compress.output=true; No rows affected (0.004 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.parallel=true; No rows affected (0.005 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.execution.engine=spark; No rows affected (0.008 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.executor.cores=1; No rows affected (0.004 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.executor.memory=3g; No rows affected (0.005 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.executor.memoryOverhead=1g; No rows affected (0.003 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.yarn.executor.memoryOverhead=1024; No rows affected (0.004 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET avro.output.codec=snappy; No rows affected (0.003 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET mapreduce.job.queuename=batchlayer; No rows affected (0.003 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> add jar hdfs:///user/admeta/udf/protobuf-hive-serde.jar; INFO : converting to local hdfs:///user/admeta/udf/protobuf-hive-serde.jar INFO : Added [/tmp/8ab71e5f-efbc-4c74-bccb-82e9475aa4e1_resources/protobuf-hive-serde.jar] to class path INFO : Added resources: [hdfs:///user/admeta/udf/protobuf-hive-serde.jar] No rows affected (0.221 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SELECT batchid, 1 FROM batch_state.import_to_event WHERE batchid IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201); INFO : Compiling command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71): SELECT batchid, 1 FROM batch_state.import_to_event WHERE batchid IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201) INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:batchid, type:bigint, comment:null), FieldSchema(name:_c1, type:int, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71); Time taken: 0.134 seconds INFO : Executing command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71): SELECT batchid, 1 FROM batch_state.import_to_event WHERE batchid IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201) INFO : Query ID = hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in parallel INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : Starting Spark Job = 5978239f-723b-4a4b-bf7e-e474da7eb6f4 INFO : Running with YARN Application = application_1538413789685_4130 INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/bin/yarn application -kill application_1538413789685_4130 INFO : Query Hive on Spark job[0] stages: INFO : 0 INFO : Status: Running (Hive on Spark job[0]) INFO : Job Progress Format CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost] INFO : 2018-10-04 15:27:43,845 Stage-0_0: 0(+1)/1 INFO : 2018-10-04 15:27:46,862 Stage-0_0: 0(+1)/1 INFO : 2018-10-04 15:27:49,878 Stage-0_0: 0(+1)/1 INFO : 2018-10-04 15:27:50,883 Stage-0_0: 1/1 Finished INFO : Status: Finished successfully in 18.06 seconds INFO : =====Spark Job[5978239f-723b-4a4b-bf7e-e474da7eb6f4] statistics===== INFO : HIVE INFO : CREATED_FILES: 1 INFO : RECORDS_OUT_0: 10 INFO : RECORDS_IN: 33056 INFO : DESERIALIZE_ERRORS: 0 INFO : Spark Job[5978239f-723b-4a4b-bf7e-e474da7eb6f4] Metrics INFO : ExecutorDeserializeTime: 2022 INFO : ExecutorRunTime: 4244 INFO : ResultSize: 2107 INFO : JvmGCTime: 194 INFO : ResultSerializationTime: 1 INFO : MemoryBytesSpilled: 0 INFO : DiskBytesSpilled: 0 INFO : BytesRead: 11526 INFO : Execution completed successfully INFO : Completed executing command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71); Time taken: 28.004 seconds INFO : OK +-------------------+------+--+ | batchid | _c1 | +-------------------+------+--+ | 2018092513420201 | 1 | | 2018092513480201 | 1 | | 2018092513540101 | 1 | | 2018092514000101 | 1 | | 2018092514060101 | 1 | | 2018092514120101 | 1 | | 2018092514180101 | 1 | | 2018092514240101 | 1 | | 2018092514300201 | 1 | | 2018092514360101 | 1 | +-------------------+------+--+ 10 rows selected (28.251 seconds) 0: jdbc:hive2://hadoop40.wopd.local:10000/def> SELECT . . . . . . . . . . . . . . . . . . . . . . .> count(*) . . . . . . . . . . . . . . . . . . . . . . .> FROM import.adserverevent ase . . . . . . . . . . . . . . . . . . . . . . .> WHERE ase.BatchId IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201); INFO : Compiling command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8): SELECT count(*) FROM import.adserverevent ase WHERE ase.BatchId IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201) INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8); Time taken: 1.17 seconds INFO : Executing command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8): SELECT count(*) FROM import.adserverevent ase WHERE ase.BatchId IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201) INFO : Query ID = hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in parallel INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : Starting Spark Job = a18751ee-97a0-4c5b-86e2-b87eeb1933dc INFO : Running with YARN Application = application_1538413789685_4130 INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/bin/yarn application -kill application_1538413789685_4130 INFO : Query Hive on Spark job[1] stages: INFO : 1 INFO : 2 INFO : Status: Running (Hive on Spark job[1]) INFO : Job Progress Format CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost] INFO : 2018-10-04 15:30:50,545 Stage-1_0: 0(+1)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:30:53,567 Stage-1_0: 0(+1)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:30:54,573 Stage-1_0: 0(+2)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:30:57,594 Stage-1_0: 0(+6)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:30:58,601 Stage-1_0: 0(+7)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:31:01,666 Stage-1_0: 0(+8)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:31:04,707 Stage-1_0: 0(+8)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:31:05,713 Stage-1_0: 0(+9)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:31:08,731 Stage-1_0: 0(+9)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:31:11,750 Stage-1_0: 0(+9)/9 Stage-2_0: 0/1 INFO : 2018-10-04 15:31:13,767 Stage-1_0: 1(+8)/9 Stage-2_0: 0/1 ERROR : Status: Failed ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask INFO : Completed executing command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8); Time taken: 30.006 seconds Error: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=3) 0: jdbc:hive2://hadoop40.wopd.local:10000/def>
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache Spark
11-07-2017
03:46 PM
Hi Jason, Thanks a lot for your continued prompt Support. Really Appreciated. I am able to deploy the Non-HA Cluster for now, which is a POC. I will take care of HA, when we decide doing a HA POC or move on to Production. Regards
... View more
11-07-2017
02:37 PM
Hi Jason, Thanks for looking into it. I have a properly Working DNS Server in Azure on the same VM where Cloudera Director is installed. - JFYI please. Cloudera Express is the selected Deployment. The Cloudera Manager is Sucessfully created on a New VM, via the Cloudera Director. Now When I am deploying the Cluster, I have selected the option of All Services and selected = Masters - 3 Nodes, Worker Nodes - 3 Nodes and GateWay - 1 Node. It failed with again "BootStrap Failed" Please check the log https://drive.google.com/file/d/1W7GB2x773SAUzu0N3y_OtrrJp24thsRv/view?usp=sharing
... View more
11-06-2017
02:08 PM
Hi, Thanks for the Work Around, so it went ahead from that step. But still the Cloudera Manager is not successfully installed. This time out of 194 steps , it went very well till 185 steps .. That was the progress it was showing.. It gave the error again with the BootStrap failed.. Please help... Below is the application log.. com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$HostInstall/5 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:35:19.078 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Found hostname for 10.0.0.5 from capabilities as wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:35:19.078 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Host installation will be performed for host name wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:35:19.979 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=585, pipeline=f2d7150c-62a1-40d0-a84a-641a5011a250, ... [2017-11-06 15:35:20.055 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for Cloudera Manager to deploy agent on 10.0.0.5] [2017-11-06 15:35:20.056 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager to deploy agent on 10.0.0.5 [2017-11-06 15:35:20.056 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:35:20.192 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure/6 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:35:20.205 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [13] [2017-11-06 15:36:02.068 -0600] INFO [task-thread-10] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2017-11-06 15:36:02.071 -0600] INFO [task-thread-10] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting [2017-11-06 15:37:23.717 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:37:23.768 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$HostInstall/5 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:37:23.821 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Found hostname for 10.0.0.5 from capabilities as wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:37:23.821 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Host installation will be performed for host name wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:37:23.952 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=600, pipeline=f2d7150c-62a1-40d0-a84a-641a5011a250, ... [2017-11-06 15:37:24.042 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for Cloudera Manager to deploy agent on 10.0.0.5] [2017-11-06 15:37:24.043 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager to deploy agent on 10.0.0.5 [2017-11-06 15:37:24.043 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:37:24.102 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure/6 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:37:24.118 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [16] [2017-11-06 15:39:13.092 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:39:13.149 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$HostInstall/5 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:39:13.197 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Found hostname for 10.0.0.5 from capabilities as wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:39:13.197 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Host installation will be performed for host name wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:39:13.292 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=615, pipeline=f2d7150c-62a1-40d0-a84a-641a5011a250, ... [2017-11-06 15:39:13.369 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for Cloudera Manager to deploy agent on 10.0.0.5] [2017-11-06 15:39:13.370 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager to deploy agent on 10.0.0.5 [2017-11-06 15:39:13.371 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:39:13.428 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure/6 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:39:13.444 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [18] [2017-11-06 15:40:02.901 -0600] INFO [task-thread-2] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models [2017-11-06 15:40:02.902 -0600] INFO [task-thread-2] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models [2017-11-06 15:40:02.914 -0600] INFO [task-thread-3] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments [2017-11-06 15:40:02.915 -0600] INFO [task-thread-3] - - - - - c.c.l.task.RefreshDeployments: Skipping refresh of deployment wopdtestenv:wopdtestcm as it is in transition or terminated. (Stage: BOOTSTRAPPING, deploymentIsNull: true) [2017-11-06 15:40:02.915 -0600] INFO [task-thread-3] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models [2017-11-06 15:40:58.650 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:40:58.701 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$HostInstall/5 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:40:58.754 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Found hostname for 10.0.0.5 from capabilities as wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:40:58.754 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Host installation will be performed for host name wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:40:58.853 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=630, pipeline=f2d7150c-62a1-40d0-a84a-641a5011a250, ... [2017-11-06 15:40:59.061 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for Cloudera Manager to deploy agent on 10.0.0.5] [2017-11-06 15:40:59.062 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager to deploy agent on 10.0.0.5 [2017-11-06 15:40:59.062 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:40:59.122 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure/6 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:40:59.133 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [20] [2017-11-06 15:41:02.071 -0600] INFO [task-thread-4] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2017-11-06 15:41:02.073 -0600] INFO [task-thread-4] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting [2017-11-06 15:42:44.700 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:42:44.750 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$HostInstall/5 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:42:44.813 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Found hostname for 10.0.0.5 from capabilities as wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:42:44.814 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Host installation will be performed for host name wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net [2017-11-06 15:42:44.913 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=645, pipeline=f2d7150c-62a1-40d0-a84a-641a5011a250, ... [2017-11-06 15:42:44.983 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for Cloudera Manager to deploy agent on 10.0.0.5] [2017-11-06 15:42:44.990 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager to deploy agent on 10.0.0.5 [2017-11-06 15:42:44.990 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-11-06 15:42:45.049 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure/6 [DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type= ... [2017-11-06 15:42:45.063 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [22] [2017-11-06 15:44:30.291 -0600] ERROR [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.b.c.BootstrapClouderaManagerAgent: Command GlobalHostInstall with ID 22 failed after 5 tries. Details: ApiCommand{id=22, name=GlobalHostInstall, startTime=Mon Nov 06 15:42:44 CST 2017, endTime=Mon Nov 06 15:44:24 CST 2017, active=false, success=false, resultMessage=Command completed with 0/1 successful subcommands, serviceRef=null, roleRef=null, hostRef=null, parent=null} [2017-11-06 15:44:30.310 -0600] ERROR [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation failed on instance '10.0.0.5' after 5 tries. at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:436) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:380) at com.cloudera.launchpad.pipeline.job.Job6.runUnchecked(Job6.java:36) at com.cloudera.launchpad.pipeline.job.Job6$$FastClassBySpringCGLIB$$54178506.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.GeneratedMethodAccessor202.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure$$EnhancerBySpringCGLIB$$6803a5e5.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:199) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:139) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2017-11-06 15:44:30.312 -0600] ERROR [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error: JobError{jobClassName=com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure, jobArguments=[DeploymentContext{environment=Environment{name='wopdtestenv', provider=InstanceProviderConfig{type='azure'}, credentials=SshCredentials{username='wopduser', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}}, deployment=Deployment{name='wopdtestcm', hostname='10.0.0.5', port=7180, username='admin', tlsEnabled=false, tlsConfigurationProperties={}, managerInstance=Optional.of(PluggableComputeInstance{ipAddress=10.0.0.5, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.0.0.5', hostAddress=Optional.of(/10.0.0.5)}, HostEndpoint{hostAddressString='51.143.99.254', hostAddress=Optional.of(/51.143.99.254)}]} Instance{virtualInstance=VirtualInstance{id='05fd6de3-6956-4ef0-b412-1a4206e40177', template=InstanceTemplate{name='wopdtemplate', type='STANDARD_DS12_V2', image='cloudera-centos-67-latest', rackId='/default', bootstrapScriptsArePresent=false, config={dataDiskSize=1024, availabilitySet=wopdtestmnavset, virtualNetworkResourceGroup=wopdtestrg, hostFqdnSuffix=cdh-cluster.internal, publicIP=Yes, dataDiskCount=5, subnetName=default, virtualNetwork=wopdtestvnet, computeResourceGroup=wopdtestrg, instanceNamePrefix=wopdtestdns, storageAccountType=Premium_LRS, managedDisks=No, useCustomManagedImage=No, networkSecurityGroupResourceGroup=wopdtestrg, networkSecurityGroup=wopdtestnsg}, tags={}, normalizeInstance=true, sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net), clouderaManagerAgentInstalled=false, customScriptPaths={RESIZE_ROOT_PARTITION=/var/lib/cloudera-director-plugins/azure-provider-2.0.0/etc/rewrite_root_disk_partition_table, PREPARE_UNMOUNTED_VOLUMES=/var/lib/cloudera-director-plugins/azure-provider-2.0.0/etc/prepare_unmounted_volumes}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2017-11-06T15:28:44.243-06:00, lastChecked=2017-11-06T15:28:44.243-06:00}}), createdExternalDatabases=[], repository='Optional.absent()', repositoryKeyUrl='Optional.absent()', enableEnterpriseTrial=Optional.of(false), unlimitedJce=Optional.absent(), krbAdminUsername='Optional.absent()', javaInstallationStrategy='AUTO', tunnelingRequired=false, cmVersion=Optional.absent()}}, PluggableComputeInstance{ipAddress=10.0.0.5, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.0.0.5', hostAddress=Optional.of(/10.0.0.5)}, HostEndpoint{hostAddressString='51.143.99.254', hostAddress=Optional.of(/51.143.99.254)}]} Instance{virtualInstance=VirtualInstance{id='05fd6de3-6956-4ef0-b412-1a4206e40177', template=InstanceTemplate{name='wopdtemplate', type='STANDARD_DS12_V2', image='cloudera-centos-67-latest', rackId='/default', bootstrapScriptsArePresent=false, config={dataDiskSize=1024, availabilitySet=wopdtestmnavset, virtualNetworkResourceGroup=wopdtestrg, hostFqdnSuffix=cdh-cluster.internal, publicIP=Yes, dataDiskCount=5, subnetName=default, virtualNetwork=wopdtestvnet, computeResourceGroup=wopdtestrg, instanceNamePrefix=wopdtestdns, storageAccountType=Premium_LRS, managedDisks=No, useCustomManagedImage=No, networkSecurityGroupResourceGroup=wopdtestrg, networkSecurityGroup=wopdtestnsg}, tags={}, normalizeInstance=true, sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(wopdtestdns-05fd6de3.cdh-cluster.internal.i1p5saafna2e3h34w5frreyphh.xx.internal.cloudapp.net), clouderaManagerAgentInstalled=false, customScriptPaths={RESIZE_ROOT_PARTITION=/var/lib/cloudera-director-plugins/azure-provider-2.0.0/etc/rewrite_root_disk_partition_table, PREPARE_UNMOUNTED_VOLUMES=/var/lib/cloudera-director-plugins/azure-provider-2.0.0/etc/prepare_unmounted_volumes}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2017-11-06T15:28:44.243-06:00, lastChecked=2017-11-06T15:28:44.243-06:00}}, Optional.of(true), true, 5, 22], jobContext=JobContext{callCountAtThisStackLevel=0, pipelineHandle='f2d7150c-62a1-40d0-a84a-641a5011a250', callStack=CallStack{items=[Item{className='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob', callCount=7}, Item{className='com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager', callCount=11}, Item{className='com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.InstallManagementServices', callCount=2}, Item{className='com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent', callCount=0}], size=4, parent=Optional.absent()}, stackLevel=4}, errorInfo=ErrorInfo{code=CM_AGENT_INSTALLATION_FAIL, properties={instanceIpAddress=10.0.0.5, retryCount=5}, causes=[]}} com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation failed on instance '10.0.0.5' after 5 tries. at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:436) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:380) at com.cloudera.launchpad.pipeline.job.Job6.runUnchecked(Job6.java:36) at com.cloudera.launchpad.pipeline.job.Job6$$FastClassBySpringCGLIB$$54178506.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.GeneratedMethodAccessor202.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure$$EnhancerBySpringCGLIB$$6803a5e5.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:199) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:139) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2017-11-06 15:44:30.320 -0600] ERROR [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.p.DatabasePipelineRunner: Pipeline 'f2d7150c-62a1-40d0-a84a-641a5011a250' failed at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure$$EnhancerBySpringCGLIB$$6803a5e5 at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent:0 [2017-11-06 15:44:30.341 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.p.s.PipelineRepositoryService: Pipeline 'f2d7150c-62a1-40d0-a84a-641a5011a250': RUNNING -> ERROR [2017-11-06 15:44:30.384 -0600] INFO [p-641a5011a250-DefaultBootstrapDeploymentJob] 2694cb90-0ed6-4373-95eb-cce0ff9eee76 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.d.DeploymentRepositoryService: Deployment 'wopdtestcm': BOOTSTRAPPING -> BOOTSTRAP_FAILED [2017-11-06 15:45:02.902 -0600] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models [2017-11-06 15:45:02.903 -0600] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models [2017-11-06 15:45:02.917 -0600] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments [2017-11-06 15:45:02.917 -0600] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Skipping refresh of deployment wopdtestenv:wopdtestcm as it is in transition or terminated. (Stage: BOOTSTRAP_FAILED, deploymentIsNull: true) [2017-11-06 15:45:02.917 -0600] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models [2017-11-06 15:46:02.073 -0600] INFO [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2017-11-06 15:46:02.075 -0600] INFO [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting [2017-11-06 15:50:02.903 -0600] INFO [task-thread-4] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models [2017-11-06 15:50:02.904 -0600] INFO [task-thread-4] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models [2017-11-06 15:50:02.920 -0600] INFO [task-thread-4] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments [2017-11-06 15:50:02.920 -0600] INFO [task-thread-4] - - - - - c.c.l.task.RefreshDeployments: Skipping refresh of deployment wopdtestenv:wopdtestcm as it is in transition or terminated. (Stage: BOOTSTRAP_FAILED, deploymentIsNull: true) [2017-11-06 15:50:02.920 -0600] INFO [task-thread-4] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models [2017-11-06 15:50:49.305 -0600] INFO [qtp646445398-281] b43a6cf5-6b03-40a2-81e4-923116a8aa5f GET /api/v10/environments/wopdtestenv - - c.c.d.a.c.c.AzureCredentials: Creating ApplicationTokenCredentials [2017-11-06 15:50:49.312 -0600] INFO [qtp646445398-281] b43a6cf5-6b03-40a2-81e4-923116a8aa5f GET /api/v10/environments/wopdtestenv - - c.c.d.a.c.c.AzureCredentials: Validating credentials by authenticating with Azure. [2017-11-06 15:50:49.330 -0600] INFO [pool-15-thread-1] - - - - - c.c.d.a.s.c.m.a.a.AuthenticationAuthority: [Correlation ID: dd954d9a-9496-46d8-bbf5-d8d6cce64960] Instance discovery was successful [2017-11-06 15:51:02.076 -0600] INFO [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2017-11-06 15:51:02.078 -0600] INFO [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
... View more
11-03-2017
12:29 PM
Hi, Yes, I did that. Please check attached the screen shot..
... View more
11-03-2017
11:14 AM
Hi, Thanks a lot for the quick Reply.. You are right, some how I overlooked the error , it has given the below error "com.cloudera.director.azure.shaded.com.microsoft.azure.CloudException: User failed validation to purchase resources. Error message: 'Legal terms have not been accepted for this item on this subscription. To accept legal terms, please go to the Azure portal (http://go.microsoft.com/fwlink/?LinkId=534873) and configure programmatic deployment for the Marketplace item or create it there for the first time' Further Actions taken.. 1) I tried and went to Azure Marketplace and enabled for all the pre images of cloudera. 2) Also I created one VM Manually using - Preimage of Cloudera Centos 6.7 (As mentioned above - "create it there for the first time") Still No Luck...and Same Error...as above..
... View more
11-02-2017
04:10 PM
Hi ,
I am able to create the Virtual Machine on Azure, Installed Cloudera Director.
While Deploying the Cloudera Manager , it failed with the Error "Bootstrap Failed" on hint it shows "Insufficient number of instances available after allocation".
Any help is on it is much appreciated...
Below is the Application.log from /var/log/cloudera-director-server
[2017-11-02 18:00:55.798 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Starting to delete VM director-154feaf6-908d-4a5a-a9f5-13aafbd85966 and its resources. [2017-11-02 18:00:55.798 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Deleting VM director-154feaf6-908d-4a5a-a9f5-13aafbd85966. [2017-11-02 18:00:55.886 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Successful delete of VM director-154feaf6-908d-4a5a-a9f5-13aafbd85966. [2017-11-02 18:00:55.886 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Deleting Storage Account 154feaf6. [2017-11-02 18:00:56.853 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Successful delete of Storage Account 154feaf6. [2017-11-02 18:00:56.854 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Deleting NIC 154feaf6. [2017-11-02 18:00:57.248 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Successful delete of NIC 154feaf6. [2017-11-02 18:00:57.248 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Deleting Public IP 154feaf6. [2017-11-02 18:01:07.445 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Successful delete of Public IP 154feaf6. [2017-11-02 18:01:07.445 -0500] INFO [RxIoScheduler-31] - - - - - c.c.d.a.c.p.AzureComputeProvider: Delete of VM director-154feaf6-908d-4a5a-a9f5-13aafbd85966 and its resources took 11 seconds. [2017-11-02 18:01:14.632 -0500] INFO [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.d.a.c.p.AzureComputeProvider: Create Virtual Machines: it took 42 seconds to create 0 out of 1 VMs with the prefix director from the group with instanceIds of: [154feaf6-908d-4a5a-a9f5-13aafbd85966]. [2017-11-02 18:01:14.632 -0500] INFO [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.d.a.c.p.AzureComputeProvider: Provisioned 0 instances out of 1. minCount is 1. [2017-11-02 18:01:14.632 -0500] INFO [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.d.a.c.p.AzureComputeProvider: Preparing to delete VMs in Resource Group wopdtestrg. Instance prefix: director. VM Ids: [154feaf6-908d-4a5a-a9f5-13aafbd85966]. [2017-11-02 18:01:14.632 -0500] INFO [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.d.a.c.p.AzureComputeProvider: Finding in Resource Group wopdtestrg with the instance prefix of director the following VMs: [154feaf6-908d-4a5a-a9f5-13aafbd85966]. [2017-11-02 18:01:14.877 -0500] INFO [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.d.a.c.p.AzureComputeProvider: Found the following VMs: []. [2017-11-02 18:01:14.878 -0500] WARN [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Error while attempting to allocate instances for group CM. Attempting to continue. com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Failed to create enough instances. at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:125) at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:616) at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:545) at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:520) at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:331) at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:220) at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196) at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$edbe32dd.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:199) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:139) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
Caused by: com.cloudera.director.spi.v2.model.exception.UnrecoverableProviderException: Failed to create enough instances. at com.cloudera.director.spi.v2.adapters.v1.ExceptionsAdapter.fromV1(ExceptionsAdapter.java:84) at com.cloudera.director.spi.v2.adapters.v1.ExceptionsAdapter.fromV1(ExceptionsAdapter.java:51) at com.cloudera.director.spi.v2.adapters.v1.ComputeProviderAdapter$1.allocate(ComputeProviderAdapter.java:125) at com.cloudera.director.spi.v2.adapters.v1.ComputeProviderAdapter$1.allocate(ComputeProviderAdapter.java:55) at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:614) ... 36 common frames omitted [2017-11-02 18:01:14.878 -0500] ERROR [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available after allocation at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:257) at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196) at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$edbe32dd.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:199) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:139) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
[2017-11-02 18:01:14.878 -0500] ERROR [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error: JobError{jobClassName=com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun, jobArguments=[VirtualInstanceGroup{name='CM', virtualInstances=[VirtualInstance{id='154feaf6-908d-4a5a-a9f5-13aafbd85966', template=InstanceTemplate{name='wopdtestcm', type='STANDARD_DS12_V2', image='cloudera-centos-67-latest', rackId='/default', bootstrapScriptsArePresent=true, config={dataDiskSize=1024, availabilitySet=wopdtestmnavnomd, virtualNetworkResourceGroup=wopdtestrg, hostFqdnSuffix=cdh-cluster.internal, publicIP=Yes, dataDiskCount=5, subnetName=default, virtualNetwork=wopdtestvnet, computeResourceGroup=wopdtestrg, instanceNamePrefix=director, storageAccountType=Premium_LRS, managedDisks=No, useCustomManagedImage=No, networkSecurityGroupResourceGroup=wopdtestrg, networkSecurityGroup=wopdtestnsg}, tags={}, normalizeInstance=true, sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE}}], serviceTypeToRoleTypes={}, roleTypesConfigs={}, minCount=1}, Environment{name='wopdtestenv', provider=InstanceProviderConfig{type='azure'}, credentials=SshCredentials{username='wopdcduser', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}}], jobContext=JobContext{callCountAtThisStackLevel=0, pipelineHandle='b59d1ce3-1951-4418-85f5-929955f5c317', callStack=CallStack{items=[Item{className='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances', callCount=0}], size=2, parent=Optional.absent()}, stackLevel=2}, errorInfo=ErrorInfo{code=INSTANCE_ALLOCATION_FAILURE, properties={minRequiredCount=1, numberOfInstancesAllocated=0, virtualInstanceGroup=CM}, causes=[]}} com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available after allocation at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:257) at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196) at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$edbe32dd.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:199) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:139) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2017-11-02 18:01:14.879 -0500] ERROR [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineRunner: Pipeline 'b59d1ce3-1951-4418-85f5-929955f5c317' failed at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$edbe32dd at com.cloudera.launchpad.bootstrap.AllocateInstances:0
[2017-11-02 18:01:14.891 -0500] INFO [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b59d1ce3-1951-4418-85f5-929955f5c317': RUNNING -> ERROR [2017-11-02 18:01:14.911 -0500] INFO [p-929955f5c317-DefaultBootstrapDeploymentJob] 4e7e437e-f090-4cf1-9bae-8bd088992af8 POST /api/v10/environments/wopdtestenv/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.d.DeploymentRepositoryService: Deployment 'wopdtestcm': BOOTSTRAPPING -> BOOTSTRAP_FAILED [2017-11-02 18:02:25.007 -0500] INFO [task-thread-10] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models [2017-11-02 18:02:25.007 -0500] INFO [task-thread-10] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models [2017-11-02 18:02:25.068 -0500] INFO [task-thread-2] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2017-11-02 18:03:24.944 -0500] INFO [task-thread-3] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2017-11-02 18:03:24.948 -0500] INFO [task-thread-3] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
... View more
Labels:
- Labels:
-
Cloudera Manager