How to create a CDH-HA using the sample aws.ha.ref...

beong · ‎09-16-2016

I have used the aws.ha.reference.conf sample config script to create on Cloudera Director's command line, and cluster was created in CDirector, but failed with bootstrap error plus the instances were created in AWS, but not populated with the specified Role services.

My current CDIR is v2.1 and CMgr is v5.7.

-- Following is copied & paste of part of the aws.ha.reference.conf and none of the Roles services were created ----

# Highly Available Cluster description
#

cluster {

# List the products and their versions that need to be installed.
# These products must have a corresponding parcel in the parcelRepositories
# configured above. The specified version will be used to find a suitable
# parcel. Specifying a version that points to more than one parcel among
# those available will result in a configuration error. Specify more granular
# versions to avoid conflicts.

products {
CDH: 5
}

#
# Optional override of CDH parcel repositories
#

#
# parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.7/"]

services: [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, HUE, OOZIE]

#
# Optional custom service configurations
# Configuration keys containing periods must be enclosed in double quotes.
#

configs {
# HDFS fencing should be set to true for HA configurations
HDFS {
dfs_ha_fencing_methods: "shell(true)"
}

# OOZIE requires a load balancer specifically for high availability.
# Director does not create or manage the load balancer.
#
# The load balancer must be configured with the IPs of the oozie servers
# after the cluster completes bootstrapping.
OOZIE {
oozie_load_balancer: "example.com:11000"
}
}

#
# High availibility configuration requires external databases to be defined for the
# Hive Metastore, Hue, and Oozie services. These databases may be configured either
# as existing external databases using the "databases" block below, or as "databaseTemplate"
# to be created on a databaseServer.
#

#
# Optional configuration for existing external database for Hive Metastore, Hue,
# and Oozie databases
#

# databases {
# HIVE {
# type: postgresql
# host: db.example.com
# port: 123
# user: hive
# password: pass
# name: hive_db
# }
# HUE {
# type: postgresql
# host: db.example.com
# port: 123
# user: hue
# password: pass
# name: hue_db
# }
# OOZIE {
# type: postgresql
# host: db.example.com
# port: 123
# user: oozie
# password: pass
# name: oozie_db
# }
# }

#
# Optional configuration for creating external database on the fly for Hive Metastore
# Hue, and Oozie databases
#

databaseTemplates: {

HIVE {
name: hivetemplate
databaseServerName: rdsmysqlprod1 # Must correspond to an external database server named above
databaseNamePrefix: hivemetastore
usernamePrefix: hive
}

HUE {
name: huetemplate
databaseServerName: rdsmysqlprod1 # Must correspond to an external database server named above
databaseNamePrefix: huedb
usernamePrefix: hue
}

OOZIE {
name: oozietemplate
databaseServerName: rdsmysqlprod1 # Must correspond to an external database server named above
databaseNamePrefix: ooziedb
usernamePrefix: oozie
}
}

#
# This reference configuration file divides the master roles into three separate virtual
# instance groups, "hdfsmasters-1", "hdfsmasters-2", "masters-1", and "masters-2". The split
# between the "hdfsmasters" and "masters" groups is due to a difference in the recovery methods
# for the different services.
#
# HDFS master roles are placed separately than non-HDFS master roles in this sample cluster
# because recovery from NameNode instance failures is performed through Cloudera Manager's
# MigrateRoles command.
# http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_nn_migrate_roles....
#

hdfsmasters-1 {
count: 2

instance: ${instances.i22x} {
tags {
group: hdfsmasters-1
}
}

roles {
HDFS: [NAMENODE, FAILOVERCONTROLLER, JOURNALNODE]
}

# NameNode nameservice, autofailover, and quorum journal name must be configured for high availability
configs {
HDFS {
NAMENODE {
dfs_federation_namenode_nameservice: hanameservice
autofailover_enabled: true
dfs_namenode_quorum_journal_name: hanameservice
}
}
}
}

hdfsmasters-2 {
count: 1

instance: ${instances.i22x} {
tags {
group: hdfsmasters-2
}
}

roles {
HDFS: [JOURNALNODE, HTTPFS] # HTTPFS role needed for HUE
}
}

masters-1 {
count: 2

instance: ${instances.i22x} {
tags {
group: masters-1
}
}

roles {
ZOOKEEPER: [SERVER]
HIVE: [HIVESERVER2, HIVEMETASTORE]
YARN: [RESOURCEMANAGER]
HBASE: [MASTER]
HUE: [HUE_SERVER]
OOZIE: [OOZIE_SERVER]
}

# Oozie plugins must be configured for high availability
configs {
OOZIE {
OOZIE_SERVER {
oozie_plugins_list: "org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService,org.apache.oozie.service.ZKJobsConcurrencyService,org.apache.oozie.service.ZKUUIDService"
}
}
}
}

masters-2 {
count: 1

instance: ${instances.i22x} {
tags {
group: master
}
}

roles {
ZOOKEEPER: [SERVER]
YARN: [JOBHISTORY]
HBASE: [HBASETHRIFTSERVER] # HBASETHRIFTSERVER role needed for HUE
}
}

workers {
count: 9
#
# Minimum number of instances required to set up the cluster.
# Fail and quit if minCount number of instances is not available in this cloud
# environment. Else, continue setting up the cluster.
#
minCount: 5

instance: ${instances.d22x} {

# Put all cluster nodes in a placement group for improved network performance
# Note: this only works for a limited set of instances
# http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html

# placementGroup: REPLACE-ME

tags {
group: worker
}
}

roles {
HDFS: [DATANODE]
YARN: [NODEMANAGER]
HBASE: [REGIONSERVER]
}

# Optional custom role configurations
# Configuration keys containing periods must be enclosed in double quotes.
# configs {
# }
}

gateways {
count: 1

instance: ${instances.t2l} {
tags {
group: gateway
}
}

roles {
HIVE: [GATEWAY]
}

# Optional custom role configurations
# Configuration keys containing periods must be enclosed in double quotes.
# configs {
# }
}
}

beong · ‎09-19-2016

Hi Cloudera:

I copied the "aws.ha.reference.conf" to my own folder in Linux OS, and after ensure that all the relevant AWS parameters are pointing my own AWS environment, I ran the below, which fails all the time. In Cloudera Director 2.1, the Roles are created, however, the services show up as unconfigured. In AWS, the instances were created as stated in the count, but the new cluster is not created in Cloudera Manager v5.7.1. What am I doing wrong and is there anything I need to configure in /etc/cloudera-director-server/application.properties file?

I need some guidance on how to create a new CDH HA cluster using the "aws.ha.reference.conf" config script.

---- Below is the command I ran with its result ----

$ cloudera-director bootstrap-remote aws.ha.cluster.conf --lp.remote.username=admin --lp.remote.password=xxxxxx --lp.remote.hostAndPort=localhost:7189
Process logs can be found at /root/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
Cloudera Director 2.1.0 initializing ...
Connecting to http://localhost:7189
Current user roles: [ROLE_ADMIN, ROLE_READONLY]
Configuration file passes all validation checks.
Creating a new environment...
Creating external database servers if configured...
Creating a new Cloudera Manager...
Creating a new CDH cluster...
* Done ...
Cloudera Manager ready.
* Starting ... done
* Requesting 10 instance(s) in 6 group(s) ............................................................................................................................................................................................................................................................................................... done
* Parallel execution of jobs failed. Total: 6 Canceled: 0 Failed: 6 ...

beong · ‎09-19-2016

Hi Cloudera:

I copied the "aws.ha.reference.conf" to my own folder in Linux OS, and after ensure that all the relevant AWS parameters are pointing my own AWS environment, I ran the below, which fails all the time. In Cloudera Director 2.1, the Roles are created, however, the services show up as unconfigured and the status is "Bootstrap failed". In AWS, the instances were created as stated in the count, but the new cluster is not created in Cloudera Manager v5.7.1. What am I doing wrong and is there anything I need to configure in /etc/cloudera-director-server/application.properties file?

I need some guidance on how to create a new CDH HA cluster using the "aws.ha.reference.conf" config script.

---- Below is the command I ran with its result ----

$ cloudera-director bootstrap-remote aws.ha.cluster.conf --lp.remote.username=admin --lp.remote.password=xxxxxx --lp.remote.hostAndPort=localhost:7189
Process logs can be found at /root/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
Cloudera Director 2.1.0 initializing ...
Connecting to http://localhost:7189
Current user roles: [ROLE_ADMIN, ROLE_READONLY]
Configuration file passes all validation checks.
Creating a new environment...
Creating external database servers if configured...
Creating a new Cloudera Manager...
Creating a new CDH cluster...
* Done ...
Cloudera Manager ready.
* Starting ... done
* Requesting 10 instance(s) in 6 group(s) ............................................................................................................................................................................................................................................................................................... done
* Parallel execution of jobs failed. Total: 6 Canceled: 0 Failed: 6 ...

beong · ‎09-19-2016

From the application.log, there is no errrors except the below WARN shows up:

[2016-09-19 22:03:32] WARN [Thread-9] - c.c.l.c.security.CipherSchemeFactory: Allowing cipher scheme to be set again, prior scheme is com.cloudera.launchpad.common.security.TripleDESCipher@182f0bf4

Mike Wilson · ‎09-19-2016

Hi Beong! Sorry for the late reply. That cipher message shouldn't be the cause of the error. I'm not super familiar with HA, so let me confer with some of my colleagues and I'll get back to you. In the meantime, is that application.log from the client or the server? I'd expect to see more errors on the server application.log.

There should be a more relevant stacktrace in the server log -- make sure it's not the job failed stacktrace, but the cause of it. It's fairly noisy -- apologies for that. You can also pastebin a large chunk of it to us (last few thousand lines) and we can review that as well.

beong · ‎09-19-2016

#
# Copyright (c) 2015 Cloudera, Inc. All rights reserved.
#

#
# Sample AWS Cloudera Director configuration file illustrating how to set up a highly available
# cluster based on the Cloudera AWS reference architecture:
# http://www.cloudera.com/content/cloudera/en/resources/library/whitepaper/cloudera-enterprise-reference-architecture-for-aws-deployments.html
#
# This configuration file is intended to demonstrate how to set up high availability clusters
# only. Please refer to the aws.reference.conf file and the Cloudera Director documentation for
# further details on how to use Cloudera Director.
#
deploymentName: CM5p
environmentName: Unravel_Test
#
# Cluster name
name: cdh571nkha-new1

#
# Cloud provider configuration (credentials, region or zone and optional default image)
#

provider {
    type: aws
    accessKeyId: "access-key"
    secretAccessKey: "secret-key"
    region: "us-east-1"
    subnetId: "subnet-5319090"
    securityGroupsIds: "sg-mysubnet"
    instanceNamePrefix: "cdir-HA"
    rootVolumeSizeGB: 200
    rootVolumeType: gp2
    associatePublicIpAddresses: true
}

#
# SSH credentials to use to connect to the instances
#
ssh {
    ## username: ec2-user # for RHEL image
    username: ec2-user
    privateKey: /root/CDirector-HA/director-scripts/configs/unravel/cdhcdirector.pem
}

#
# A list of instance types to use for group of nodes or management services
#

instances {

    m44x {
      type: m4.4xlarge   # requires an HVM AMI
      image: "ami-6d1c30909"
      # iamProfileName: "cdh_create_cluster_role"
      tags {
          owner: "12345678901"
      }
      bootstrapScript: """#!/bin/sh

# If the exit code is not zero Cloudera Director will automatically retry

echo 'Hello World!'
exit 0

"""
    }

   r32x {
        type: r3.2xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    m42x {
        type: m4.2xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    c34x {
        type: c3.4xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    c38x {
        type: c3.8xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    c44x {
        type: c4.4xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    i2x {
        type: i2.xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    i22x {
        type: i2.2xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    d2x {
        type: d2.xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    d22x {
        type: d2.2xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    d24x {
        type: d2.4xlarge
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }

    t2l {   # only suitable as a gateway
        type: t2.large
        image: "ami-6d1c30909"

        tags {
            owner: "12345678901"
        }
    }
}

databaseServers {
    existingmysql1 {
        type: mysql
        host: ip-10-0-0-123
        port: 3306
        user: root
        password: ZL2wIf1s
    }
}

cloudera-manager {

    instance: ${instances.m42x} {
        tags {
            # add any additional tags as needed
            application: "Cloudera Manager 5"
        }
    }
    username: admin
    password: admin

#    databaseTemplates {
#        CLOUDERA_MANAGER {
#            name: cmtemplate
#            databaseServerName: existingmysql1 # Must correspond to an external database server named above
#            databaseNamePrefix: scm
#            usernamePrefix: cmadmin
#        }
#
#        ACTIVITYMONITOR {
#            name: amontemplate
#            databaseServerName: existingmysql1 # Must correspond to an external database server named above
#            databaseNamePrefix: amon
#            usernamePrefix: amadmin
#        }
#
#        REPORTSMANAGER {
#            name: rmantemplate
#            databaseServerName: existingmysql1 # Must correspond to an external database server named above
#            databaseNamePrefix: rman
#            usernamePrefix: rmadmin
#        }
#
#        NAVIGATOR {
#            name: navtemplate
#            databaseServerName: existingmysql1 # Must correspond to an external database server named above
#            databaseNamePrefix: nav
#            usernamePrefix: nadmin
#        }
#
#        # Added in Cloudera Manager 5.2+
#        NAVIGATORMETASERVER {
#            name: navmetatemplate
#            databaseServerName: existingmysql1 # Must correspond to an external database server named above
#            databaseNamePrefix: navmeta
#            usernamePrefix: nmadmin
#        }
#    }

    #
    # Configuration to override Cloudera Manager package repositories
    #
    repository: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.7.3/"
    repositoryKeyUrl: "http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera"
}

#
# Highly Available Cluster description
#
cluster {

    products {
      CDH: 5
    }

    parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.7.1/"]

    services: [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, HUE, OOZIE]

    configs {
        # HDFS fencing should be set to true for HA configurations
        HDFS {
            dfs_ha_fencing_methods: "shell(true)"
        }
    }

    # Creating external database on the fly for Hive Metastore, Hue, and Oozie databases

    databaseTemplates: {

        HIVE {
            name: hivetemplate
            databaseServerName: existingmysql1
            databaseNamePrefix: hivemetastore
            usernamePrefix: hive
        }

        HUE {
            name: huetemplate
            databaseServerName: existingmysql1
            databaseNamePrefix: huedb
            usernamePrefix: hue
        }

        OOZIE {
            name: oozietemplate
            databaseServerName: existingmysql1
            databaseNamePrefix: ooziedb
            usernamePrefix: oozie
        }
    }

    ## Spinning up different types of CDH -HA nodes ##

    hdfsmasters-1 {
        count: 2

        instance: ${instances.m42x} {
            tags {
                group: hdfsmasters-1
            }
        }

        roles {
            HDFS: [NAMENODE, FAILOVERCONTROLLER, JOURNALNODE]
        }

        # NameNode nameservice, autofailover, and quorum journal name must be configured for high availability
        configs {
            HDFS {
                NAMENODE {
                    dfs_federation_namenode_nameservice: hanameservice
                    autofailover_enabled: true
                    dfs_namenode_quorum_journal_name: hanameservice
                }
            }
        }
    }

    hdfsmasters-2 {
        count: 1

        instance: ${instances.m42x} {
            tags {
                group: hdfsmasters-2
            }
        }

        roles {
            HDFS: [JOURNALNODE, HTTPFS] # HTTPFS role needed for HUE
        }
    }

    masters-1 {
        count: 2

        instance: ${instances.m42x} {
            tags {
                group: masters-1
            }
        }

        roles {
            ZOOKEEPER: [SERVER]
            HIVE: [HIVESERVER2, HIVEMETASTORE]
            YARN: [RESOURCEMANAGER]
            HBASE: [MASTER]
            HUE: [HUE_SERVER]
            OOZIE: [OOZIE_SERVER]
        }

        # Oozie plugins must be configured for high availability
        configs {
            OOZIE {
                OOZIE_SERVER {
                    oozie_plugins_list: "org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService,org.apache.oozie.service.ZKJobsConcurrencyService,org.apache.oozie.service.ZKUUIDService"
                }
            }
        }
    }

    masters-2 {
        count: 1

        instance: ${instances.m42x} {
            tags {
                group: master
            }
        }

        roles {
            ZOOKEEPER: [SERVER]
            YARN: [JOBHISTORY]
            HBASE: [HBASETHRIFTSERVER] # HBASETHRIFTSERVER role needed for HUE
        }
    }

    workers {
      count: 3
      minCount: 3

      instance: ${instances.c34x} {
        ## placementGroup: "c3.4xlarge"
        tags {
          group: worker
        }
      }

      roles {
        HDFS: [DATANODE]
        YARN: [NODEMANAGER]
        HBASE: [REGIONSERVER]
      }

      # Optional custom role configurations
      # Configuration keys containing periods must be enclosed in double quotes.
      # configs {
      # }
    }

    gateways {
      count: 1

      instance: ${instances.m44x} {
        tags {
          group: gateway
        }
      }

      roles {
        HIVE: [GATEWAY]
      }

      # Optional custom role configurations
      # Configuration keys containing periods must be enclosed in double quotes.
      # configs {
      # }
    }
}

beong · ‎09-19-2016

Mike:

The previous code is the content of my aws.ha.cluster.conf file that I ran. Can you spot what is wrong with it?

Also, the application.log is when I ran the command. I guess that would be the server side.

If you can get any of your peer to help out to get the config file run successfully, that would be awesome!

Mike Wilson · ‎09-20-2016

We definitely need some more log info to figure this one out. The application.log should be located at /var/log/cloudera-director-server/application.log. Is that the one you've been pasting us?

beong · ‎09-20-2016

Here is the server side /var/log/cloudera-director-server/application.log - pls let me know if you can guide me in the fixing the config file.

[2016-09-20 16:09:53] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:09:53] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:09:58] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:09:58] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:02] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:06] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:06] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:10] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:11] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:19] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:19] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:23] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:23] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:26] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:26] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:39] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:39] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:39] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:40] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:40] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:41] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:41] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:46] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:46] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:50] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:51] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:53] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:02] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:02] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:02] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-149] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c/child-00000-1e67b9d4-1fd9-49b8-801a-ccbfda53195d suspended due to failure
[2016-09-20 16:11:14] ERROR [pipeline-thread-149] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c/child-00000-1e67b9d4-1fd9-49b8-801a-ccbfda53195d' failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-150] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7/child-00000-8ba2490e-c661-4a53-8adf-57749d7e5607 suspended due to failure
[2016-09-20 16:11:14] ERROR [pipeline-thread-150] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7/child-00000-8ba2490e-c661-4a53-8adf-57749d7e5607' failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-147] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-147] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:14] ERROR [pipeline-thread-147] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c' failed
[2016-09-20 16:11:14] INFO  [pipeline-thread-147] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c': RUNNING -> ERROR
[2016-09-20 16:11:15] ERROR [pipeline-thread-145] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:15] ERROR [pipeline-thread-145] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:15] ERROR [pipeline-thread-145] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7' failed
[2016-09-20 16:11:15] INFO  [pipeline-thread-145] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7': RUNNING -> ERROR
[2016-09-20 16:11:15] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-152] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00001-2f753dbf-e423-46ab-8a56-ec996960abee suspended due to failure
[2016-09-20 16:11:19] ERROR [pipeline-thread-152] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00001-2f753dbf-e423-46ab-8a56-ec996960abee' failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-151] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00000-cbf7dfb4-0d9c-43d1-a570-68e1a54a8b9d suspended due to failure
[2016-09-20 16:11:19] ERROR [pipeline-thread-151] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00000-cbf7dfb4-0d9c-43d1-a570-68e1a54a8b9d' failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-146] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-146] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:19] ERROR [pipeline-thread-146] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3' failed
[2016-09-20 16:11:19] INFO  [pipeline-thread-146] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3': RUNNING -> ERROR
[2016-09-20 16:11:22] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:22] ERROR [pipeline-thread-154] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793/child-00000-589a57d1-4f70-4733-8e23-fb1ec7f48d40 suspended due to failure
[2016-09-20 16:11:22] ERROR [pipeline-thread-154] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793/child-00000-589a57d1-4f70-4733-8e23-fb1ec7f48d40' failed
[2016-09-20 16:11:23] ERROR [pipeline-thread-143] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:23] ERROR [pipeline-thread-143] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:23] ERROR [pipeline-thread-143] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793' failed
[2016-09-20 16:11:23] INFO  [pipeline-thread-143] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793': RUNNING -> ERROR
[2016-09-20 16:11:34] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:34] ERROR [pipeline-thread-157] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00000-36b31044-6873-4fe5-a2b1-de8fbdb92cc2 suspended due to failure
[2016-09-20 16:11:34] ERROR [pipeline-thread-157] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00000-36b31044-6873-4fe5-a2b1-de8fbdb92cc2' failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-156] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00002-52f2769e-9f36-4e41-ae0a-75e09e00008e suspended due to failure
[2016-09-20 16:11:35] ERROR [pipeline-thread-156] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00002-52f2769e-9f36-4e41-ae0a-75e09e00008e' failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-158] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00001-14c1f6d1-6da6-4994-b810-dd6ddde9a569 suspended due to failure
[2016-09-20 16:11:35] ERROR [pipeline-thread-158] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00001-14c1f6d1-6da6-4994-b810-dd6ddde9a569' failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-148] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-148] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:35] ERROR [pipeline-thread-148] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373' failed
[2016-09-20 16:11:35] INFO  [pipeline-thread-148] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373': RUNNING -> ERROR
[2016-09-20 16:11:47] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:47] ERROR [pipeline-thread-153] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00001-534e4a8d-b4c0-4424-b8b2-d0f08342947e suspended due to failure
[2016-09-20 16:11:47] ERROR [pipeline-thread-153] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00001-534e4a8d-b4c0-4424-b8b2-d0f08342947e' failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-155] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00000-82f63522-031c-4019-89f0-b3acf9efb701 suspended due to failure
[2016-09-20 16:11:48] ERROR [pipeline-thread-155] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00000-82f63522-031c-4019-89f0-b3acf9efb701' failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-144] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-144] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:48] ERROR [pipeline-thread-144] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616' failed
[2016-09-20 16:11:48] INFO  [pipeline-thread-144] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616': RUNNING -> ERROR
[2016-09-20 16:11:48] ERROR [pipeline-thread-142] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-142] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:48] ERROR [pipeline-thread-142] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b' failed
[2016-09-20 16:11:48] INFO  [pipeline-thread-142] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b': RUNNING -> ERROR

beong · ‎09-20-2016

Mike,

How can I attached the /var/log/cloudera-director-server/application.log file to this post? It seems I cannot attached a text file.

In the case I cannot, I have grep all the ERROR from /var/log/cloudera-director-server/application.log as follows:

[2016-09-20 16:09:53] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:09:53] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:09:58] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:09:58] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:02] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:06] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:06] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:10] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:11] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:14] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:19] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:19] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:23] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:23] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:26] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:26] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:27] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:39] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:39] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:39] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:40] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:40] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:41] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:41] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:46] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:46] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:50] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:51] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:10:53] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:02] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:02] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:02] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-149] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-149] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c/child-00000-1e67b9d4-1fd9-49b8-801a-ccbfda53195d suspended due to failure
[2016-09-20 16:11:14] ERROR [pipeline-thread-149] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c/child-00000-1e67b9d4-1fd9-49b8-801a-ccbfda53195d' failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-150] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-150] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7/child-00000-8ba2490e-c661-4a53-8adf-57749d7e5607 suspended due to failure
[2016-09-20 16:11:14] ERROR [pipeline-thread-150] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7/child-00000-8ba2490e-c661-4a53-8adf-57749d7e5607' failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-147] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:14] ERROR [pipeline-thread-147] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:14] ERROR [pipeline-thread-147] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c' failed
[2016-09-20 16:11:14] INFO  [pipeline-thread-147] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00004-9934ae87-37b2-4b56-99c7-1ff793b5972c': RUNNING -> ERROR
[2016-09-20 16:11:15] ERROR [pipeline-thread-145] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:15] ERROR [pipeline-thread-145] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:15] ERROR [pipeline-thread-145] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7' failed
[2016-09-20 16:11:15] INFO  [pipeline-thread-145] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00002-e4fb4729-e05b-47d9-bafd-d9dddd09aed7': RUNNING -> ERROR
[2016-09-20 16:11:15] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-152] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-152] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00001-2f753dbf-e423-46ab-8a56-ec996960abee suspended due to failure
[2016-09-20 16:11:19] ERROR [pipeline-thread-152] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00001-2f753dbf-e423-46ab-8a56-ec996960abee' failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-151] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-151] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00000-cbf7dfb4-0d9c-43d1-a570-68e1a54a8b9d suspended due to failure
[2016-09-20 16:11:19] ERROR [pipeline-thread-151] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3/child-00000-cbf7dfb4-0d9c-43d1-a570-68e1a54a8b9d' failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-146] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:19] ERROR [pipeline-thread-146] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:19] ERROR [pipeline-thread-146] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3' failed
[2016-09-20 16:11:19] INFO  [pipeline-thread-146] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00003-9820c23b-ff0a-4eb9-9a66-07beca609da3': RUNNING -> ERROR
[2016-09-20 16:11:22] ERROR [pipeline-thread-154] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:22] ERROR [pipeline-thread-154] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793/child-00000-589a57d1-4f70-4733-8e23-fb1ec7f48d40 suspended due to failure
[2016-09-20 16:11:22] ERROR [pipeline-thread-154] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793/child-00000-589a57d1-4f70-4733-8e23-fb1ec7f48d40' failed
[2016-09-20 16:11:23] ERROR [pipeline-thread-143] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:23] ERROR [pipeline-thread-143] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:23] ERROR [pipeline-thread-143] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793' failed
[2016-09-20 16:11:23] INFO  [pipeline-thread-143] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00000-a8f30c95-d228-4fea-8cf2-6c25e1aad793': RUNNING -> ERROR
[2016-09-20 16:11:34] ERROR [pipeline-thread-157] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:34] ERROR [pipeline-thread-157] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00000-36b31044-6873-4fe5-a2b1-de8fbdb92cc2 suspended due to failure
[2016-09-20 16:11:34] ERROR [pipeline-thread-157] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00000-36b31044-6873-4fe5-a2b1-de8fbdb92cc2' failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-156] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-156] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00002-52f2769e-9f36-4e41-ae0a-75e09e00008e suspended due to failure
[2016-09-20 16:11:35] ERROR [pipeline-thread-156] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00002-52f2769e-9f36-4e41-ae0a-75e09e00008e' failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-158] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-158] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00001-14c1f6d1-6da6-4994-b810-dd6ddde9a569 suspended due to failure
[2016-09-20 16:11:35] ERROR [pipeline-thread-158] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373/child-00001-14c1f6d1-6da6-4994-b810-dd6ddde9a569' failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-148] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:35] ERROR [pipeline-thread-148] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:35] ERROR [pipeline-thread-148] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373' failed
[2016-09-20 16:11:35] INFO  [pipeline-thread-148] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00005-ddd11da1-c30e-477a-b1ec-7b8a0b9bb373': RUNNING -> ERROR
[2016-09-20 16:11:47] ERROR [pipeline-thread-153] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:47] ERROR [pipeline-thread-153] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00001-534e4a8d-b4c0-4424-b8b2-d0f08342947e suspended due to failure
[2016-09-20 16:11:47] ERROR [pipeline-thread-153] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00001-534e4a8d-b4c0-4424-b8b2-d0f08342947e' failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-155] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-155] - c.c.l.p.DatabasePipelineRunner: Pipeline 2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00000-82f63522-031c-4019-89f0-b3acf9efb701 suspended due to failure
[2016-09-20 16:11:48] ERROR [pipeline-thread-155] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616/child-00000-82f63522-031c-4019-89f0-b3acf9efb701' failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-144] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-144] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:48] ERROR [pipeline-thread-144] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616' failed
[2016-09-20 16:11:48] INFO  [pipeline-thread-144] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b/child-00001-6254aadd-e1bc-47cd-b0b1-7bd886dc4616': RUNNING -> ERROR
[2016-09-20 16:11:48] ERROR [pipeline-thread-142] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
[2016-09-20 16:11:48] ERROR [pipeline-thread-142] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
[2016-09-20 16:11:48] ERROR [pipeline-thread-142] - c.c.l.p.DatabasePipelineRunner: Pipeline '2179431f-5d45-433c-90be-a1301f77149b' failed
[2016-09-20 16:11:48] INFO  [pipeline-thread-142] - c.c.l.p.s.PipelineRepositoryService: Pipeline '2179431f-5d45-433c-90be-a1301f77149b': RUNNING -> ERROR

How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera