Reply
New Contributor
Posts: 4
Registered: ‎05-10-2016

Director 2.0 failing on "CreateHiveDatabase" w/ bootstrap-remote

I'm new to Cloudera and am really out of ideas on this...

 

I've been using the bootstrap-remote command to launch my instances and initially started out with using this CentOS6 community AMI: ami-05cf2265. Turns out this AMI didn't have all the necessary cloud hardware packages installed so I installed them per our Cloudera Support. I then created an AMI image from that and tried to bootstrap-remote using my new AMI id, but it keeps failing with "Command Create Hive Database is not currently available for execution" during the bootstrap process. I was able to bootstrap just fine with the original image but the root partition does not resize properly without the other packages installed. Can someone look at my logs and point me in the right direction? 

 

Here's my very simple cluster .conf script:

#
# Copyright (c) 2014 Cloudera, Inc. All rights reserved.
#

#
# Simple AWS Cloudera Director configuration file with automatic role assignments
# that works as expected if you use a single instance type for all cluster nodes
#

#
# Cluster name
#

name: EC2

#
# Cloud provider configuration (credentials, region or zone and optional default image)
#

provider {
    type: aws

    accessKeyId: removed
    secretAccessKey: removed
    region: us-west-1
    subnetId: subnet-removed
    securityGroupsIds: removed
    instanceNamePrefix: cloudera-director

    #
    # Specify a size for the root volume (in GBs). Cloudera Director will automatically expand the
    # filesystem so that you can use all the available disk space for your application
    # See: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage_expand_partition.html
    #

    # rootVolumeSizeGB: 100 # defaults to 50 GB if not specified

    #
    # Specify the type of the EBS volume used for the root partition. Defaults to gp2
    # See: http://aws.amazon.com/ebs/details/
    #

    # rootVolumeType: standard

    #
    # Whether to associate a public IP address with instances or not. If this is false
    # we expect instances to be able to access the internet using a NAT instance
    #
    # Currently the only way to get optimal S3 data transfer performance is to assign
    # public IP addresses to your instances and not use NAT (public subnet type of setup)
    #
    # See: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-ip-addressing.html
    #

    associatePublicIpAddresses: true

}

#
# SSH credentials to use to connect to the instances
#

ssh {
    username: centos # for RHEL image
    privateKey: removed # with an absolute path to .pem file
}

#
# A list of instance types to use for group of nodes or management services
#

instances {

    m4x{
        type: m4.xlarge   # requires an HVM AMI
        image: ami-removed

        tags {
            owner: ${?USER}
        }

        bootstrapScript: """#!/bin/sh

# This is an embedded bootstrap script that runs as root and can be used to customize
# the instances immediately after boot and before any other Cloudera Director action

# Adding entry of localhost info into /etc/hosts

ip=`hostname -i`
fqdn=`hostname`
hostname=`hostname -s`

echo $ip $fqdn $hostname >> /etc/hosts

"""

        #
        # Flag indicating whether to normalize the instance. Not setting normalization here implies that your
        # bootstrap script will take care of normalization. This is an advanced configuration that will require
        # assistance from Cloudera support.
        #
        # Normalization includes:
        #   downloading and installing packages
        #   minimizing swappiness
        #   increasing the maximun number of open files
        #   mounting ephemeral disks
        #   resizing the root partition.
        #
        # Defaults to true
        #

        # normalizeInstance: true

    }
    
}

#
# Configuration for Cloudera Manager. Cloudera Director can use an existing instance
# or bootstrap everything from scratch for a new cluster
#

cloudera-manager {

    instance: ${instances.m4x} {
        tags {
            application: "Cloudera Manager 5"
        }
    }

    #
    # Automatically activate 60-Day Cloudera Enterprise Trial
    #

    enableEnterpriseTrial: true


    repository: "http://archive.cloudera.com/cm5/redhat/5/x86_64/cm/5.5.0"
    repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/5/x86_64/cm/RPM-GPG-KEY-cloudera"

    configs {
        # CLOUDERA_MANAGER corresponds to the Cloudera Manager Server configuration options
        CLOUDERA_MANAGER {
            enable_api_debug: false
            custom_banner_html: "Managed by Cloudera Director"
        }
        SERVICEMONITOR {
	        firehose_non_java_memory_bytes: 1610612736
        }
        HOSTMONITOR {
	        firehose_non_java_memory_bytes: 1610612736
        }
    }
}

#
# Cluster description
#

cluster {

    # List the products and their versions that need to be installed.
    # These products must have a corresponding parcel in the parcelRepositories
    # configured above. The specified version will be used to find a suitable
    # parcel. Specifying a version that points to more than one parcel among
    # those available will result in a configuration error. Specify more granular
    # versions to avoid conflicts.

    products {
      CDH: 5 # includes Impala and Spark
    }
    parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.5.0/"]
    services: [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, IMPALA, SPARK_ON_YARN]

    nodes {
      count: 3
      instance: ${instances.m4x}
    }

    postCreateScripts: ["""#!/bin/sh

# This is an embedded post creation script script that runs as root and can be used to
# customize the cluster after it has been created.

# If the exit code is not zero Cloudera Director will fail

# Adding entry of localhost info into /etc/hosts

ip=`hostname -i`
fqdn=`hostname`
hostname=`hostname -s`

echo $ip $fqdn $hostname >> /etc/hosts
   """,
    """#!/bin/sh

 Additionally, multiple post creation scripts can be supplied.  They will run in the
 order they are listed here.

echo 'Hello again!'
exit 0
    """]
}

Application.log

[2016-05-10 16:21:39] INFO  [pipeline-thread-63] - c.c.l.bootstrap.cluster.AddServices: Creating and configuring services [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, IMPALA, SPARK_ON_YARN]
[2016-05-10 16:21:39] INFO  [pipeline-thread-63] - c.c.launchpad.pipeline.AbstractJob: Creating cluster services
[2016-05-10 16:21:40] INFO  [pipeline-thread-63] - c.c.launchpad.pipeline.AbstractJob: Automatically assigning roles to instances
[2016-05-10 16:21:41] INFO  [pipeline-thread-63] - c.c.launchpad.pipeline.AbstractJob: Automatically configuring services and roles
[2016-05-10 16:21:42] INFO  [pipeline-thread-63] - c.c.launchpad.pipeline.AbstractJob: Applying custom configurations of services
[2016-05-10 16:21:42] INFO  [pipeline-thread-63] - c.c.launchpad.pipeline.AbstractJob: Configuring HIVE database
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=35015, pipeline=517a9481-626d-44d0-bee1-3b549e12ec95 ...
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.pipeline.util.PipelineRunner: >> FirstRunCluster/2 [Deployment{name='EC2 Deployment', hostname='172.17.3.43', port=7180, username='admin', managerInsta ...
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.pipeline.util.PipelineRunner: >> PrepareCluster/2 [Deployment{name='EC2 Deployment', hostname='172.17.3.43', port=7180, username='admin', managerInsta ...
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.launchpad.pipeline.AbstractJob: Preparing cluster EC2
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.launchpad.pipeline.AbstractJob: Creating Hive Metastore Database
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/3 [51, Deployment{name='EC2 Deployment', hostname='172.17.3.43', port=7180, username='admin', managerI ...
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.b.UnboundedWaitForApiCommand: Collecting and downloading diagnostic data
[2016-05-10 16:21:43] INFO  [pipeline-thread-63] - c.c.l.b.ClouderaManagerLogRetriever: Diagnostic data collection initiated, command ID 52
[2016-05-10 16:22:44] INFO  [pipeline-thread-63] - c.c.l.b.UnboundedWaitForApiCommand: Attempting download of diagnostic data
[2016-05-10 16:22:44] WARN  [pipeline-thread-63] - c.c.l.b.UnboundedWaitForApiCommand: Failed to download diagnostic data
org.apache.http.client.ClientProtocolException: null
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909) ~[httpclient-4.2.jar!/:4.2]
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) ~[httpclient-4.2.jar!/:4.2]
	at com.cloudera.launchpad.common.http.HttpDownloader.download(HttpDownloader.java:188) ~[launchpad-common-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.common.http.HttpDownloader.download(HttpDownloader.java:147) ~[launchpad-common-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.collectAndDownloadDiagnosticData(UnboundedWaitForApiCommand.java:132) [launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:79) [launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:41) [launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) [spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) [metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) [spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$59f51200.runUnchecked(<generated>) [spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Caused by: org.apache.http.HttpException: Unable to establish route: planned = {}->http://ip-172-17-3-43:7180; current = {}->http://172.17.3.43:7180
	at org.apache.http.impl.client.DefaultRequestDirector.establishRoute(DefaultRequestDirector.java:842) ~[httpclient-4.2.jar!/:4.2]
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) ~[httpclient-4.2.jar!/:4.2]
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) ~[httpclient-4.2.jar!/:4.2]
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ~[httpclient-4.2.jar!/:4.2]
	... 38 common frames omitted
[2016-05-10 16:22:44] INFO  [pipeline-thread-63] - c.c.l.b.UnboundedWaitForApiCommand: Diagnostic data available at http://ip-172-17-3-43:7180/cmf/command/52/download
[2016-05-10 16:22:44] ERROR [pipeline-thread-63] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateHiveDatabase with ID 51 failed. Details: ApiCommand{id=51, name=CreateHiveDatabase, startTime=Tue May 10 16:21:45 PDT 2016, endTime=Tue May 10 16:21:45 PDT 2016, active=false, success=false, resultMessage=Command Create Hive Metastore Database is not currently available for execution., serviceRef=ApiServiceRef{peerName=null, clusterName=EC2, serviceName=CD-HIVE-GbdvLGqT}, roleRef=null, hostRef=null, parent=null}
[2016-05-10 16:22:44] ERROR [pipeline-thread-63] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager 'CreateHiveDatabase' command execution failed: Command Create Hive Metastore Database is not currently available for execution.
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:87) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:41) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$59f51200.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
[2016-05-10 16:22:44] ERROR [pipeline-thread-63] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager 'CreateHiveDatabase' command execution failed: Command Create Hive Metastore Database is not currently available for execution.
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:87) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:41) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$59f51200.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
[2016-05-10 16:22:44] ERROR [pipeline-thread-63] - c.c.l.p.DatabasePipelineRunner: Pipeline '517a9481-626d-44d0-bee1-3b549e12ec95' failed
	at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$59f51200
	at com.cloudera.launchpad.bootstrap.cluster.firstrun.PrepareCluster:0

[2016-05-10 16:22:44] INFO  [pipeline-thread-63] - c.c.l.p.s.PipelineRepositoryService: Pipeline '517a9481-626d-44d0-bee1-3b549e12ec95': RUNNING -> ERROR
[2016-05-10 16:22:44] INFO  [pipeline-thread-63] - c.c.l.d.ClusterRepositoryService: Cluster 'EC2': BOOTSTRAPPING -> BOOTSTRAP_FAILED

Thanks in advance for any insight.

Highlighted
Expert Contributor
Posts: 82
Registered: ‎02-24-2016

Re: Director 2.0 failing on "CreateHiveDatabase" w/ bootstrap-remote

Did you find any fix
Cloudera Employee
Posts: 47
Registered: ‎10-28-2014

Re: Director 2.0 failing on "CreateHiveDatabase" w/ bootstrap-remote

kbhaitanya,

 

The CreateHiveDatabase issue is most commonly due to a lack of reverse dns lookup of the CM host from the Director host. This issue was fixed in 2.1.0. If you have experienced a similar issue in Director >= 2.1.0 then please start a new topic with the details of your problem so we can investigate.

 

Thanks!

David

Announcements