Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Expert Contributor
Unfortunately, these errors only tell us that the bootstrapping pipeline failed, but they don't tell us how. We need more of the log to be able to determine the failure. I would suggest using pastebin again.

I will also note that the access level of the AWS IAM account being used won't affect the errors seen in the previous log -- these would require modification of security groups or potentially re-provisioning of AWS keys (depending on the error).

If you paste us some more of your logs, though, I can take another look.

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Explorer

Mike:

 

AWS IAM account is not the issue.  I tested the key with admin privileges and I'm still getting issue. Now, I wondering Cloudera Director v2.1 has a bug with regards to executing sample config file given to customer.

 

Reason is I tested by running a simple config file, and faced the same issue.

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Explorer

Mike:

 

I tried another config file called "aws.enableha.minimal.conf" with code as follows and it failed with bootstrap errors as well:

#
# (c) Copyright 2015 Cloudera, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
deploymentName: CM5p
environmentName: Unravel_Test
#
# Cluster name
#
name: cdh571-Enable-HA-Minimal

#
# Cloud provider configuration (credentials, region or zone and optional default image)
#

provider {
    type: aws
    accessKeyId: "AAAAAAAAAAAAAA"
    secretAccessKey: "bbbbbbbbbbbbbbbbbbbbbbb"
    region: "us-east-1"
    subnetId: "subnet-531cf524"
    securityGroupsIds: "sg-de0c92bb"
    instanceNamePrefix: "cdir-HA"
    rootVolumeSizeGB: 200
    rootVolumeType: gp2
    associatePublicIpAddresses: true
}

#
# SSH credentials to use to connect to the instances
#

ssh {
    username: ec2-user # for RHEL image
    privateKey: /root/CDirector-HA/director-scripts/configs/unravel/cdhcdirector.pem
}

#
# A list of instance types to use for group of nodes or management services
#

instances {
    m42x {
        type: m4.2xlarge
        image: "ami-6d1c2007"
        # iamProfileName: "cdh_create_cluster_role"
        tags {
          purpose: "cdh571-Enable-HA-Minimal"
          owner: "xxx"
        }

        bootstrapScript: """#!/bin/sh

# This is an embedded bootstrap script that runs as root and can be used to customize
# the instances immediately after boot and before any other Cloudera Director action

# If the exit code is not zero Cloudera Director will automatically retry

echo 'Hello World!'
exit 0

"""
    }

    i2x {
      type: i2.xlarge
      image: "ami-6d1c2007"

      tags {
        purpose: "cdh571-Enable-HA-Minimal"
        owner: "xxx"
      }
    }

    d2x {
      type: d2.xlarge
      image: "ami-6d1c2007"

      tags {
        purpose: "cdh571-Enable-HA-Minimal"
        owner: "xxx"
      }
    }
}

#
# Configuration for Cloudera Manager. Cloudera Director can use an existing instance
# or bootstrap everything from scratch for a new cluster
#

#
# Configuration for Cloudera Manager. Cloudera Director can use an existing instance
# or bootstrap everything from scratch for a new cluster
#

cloudera-manager {

    instance: ${instances.m42x} {
        tags {
            application: "Cloudera Manager 5"
        }
    }
    username: admin
    password: admin
    # enableEnterpriseTrial: true
}

#
# Cluster description
#

cluster {
    products {
      CDH: 5
    }

    services: [HDFS, ZOOKEEPER]

    masters-1 {
      count: 1
      instance: ${instances.i2x} {
        tags {
          group: masters-1
        }
      }
      roles {
        HDFS: [NAMENODE, JOURNALNODE]
        ZOOKEEPER: [SERVER]
      }
    }

    masters-2 {
      count: 1
      instance: ${instances.i2x} {
        tags {
          group: masters-2
        }
      }
      roles {
        HDFS: [SECONDARYNAMENODE, JOURNALNODE]
        ZOOKEEPER: [SERVER]
      }
    }

    masters-3 {
      count: 1
      instance: ${instances.i2x} {
        tags {
          group: masters-3
        }
      }
      roles {
        HDFS: [JOURNALNODE]
        ZOOKEEPER: [SERVER]
      }
    }

    workers {
      count: 3
      instance: ${instances.d2x} {
        tags {
          group: workers
        }
      }
      roles {
        HDFS: [DATANODE]
      }
    }
}

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Expert Contributor

Without seeing more logs, I'd still hazard that this is a key or security group error, which can only be fixed from the AWS account. If you send us the logs, we'll be able to comment more, but if the error is similar to before, you'll need to inspect your AWS infrastructure settings. I'd verify that connectivity can be established using this security group and VPC setting.

 

Here are some docs on setting these up properly: https://www.cloudera.com/documentation/director/latest/topics/director_aws_setup_client.html You may be able to use this guide to compare against your current security group/VPC settings and ensure that they match up as expected.

Highlighted

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Explorer

Mike:

 

I have pasted the /var/log/cloudera-server/application.log in pastebin.com - http://pastebin.com/GmiAH9P9

 

Search for below topic.

How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

 

Let me know if you found anything...

 

Thank you and desperate for some help that can make the config files to execute successfully in my CDir v2.1/CMgr v.5.7.1 environment.

 

The other option is copy my "aws.enableha.minimal.conf" code that was attached to this thread, and see if you can create a cluster with it.

 

Regards,

Beatrice

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Expert Contributor
Okay, this is looking like the same issue from before, unfortunately. Could you take a look at your networking/security group configuration and compare against that guide from before? Let me know if any of that is different.

One thing I noticed, though, is that your key is definitely noted as found in the log, so I'd say it's unlikely that it's a key issue.

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Expert Contributor

Please include a pastebin link to the entire config file (with sensitive information redacted) as well as the entire Cloudera Director server log.

 

Have you already successfully bootstrapped a simple non-HA cluster? That could help iron out any problems related to your network configuration before tackling the added complexity of HA.

 

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Explorer

Jadair:

 

The full config file is copied and pasted in the post in "‎09-22-2016 01:41 PM" and can you copied that your cluster and try to create it.  

 

I have already paste the server's application.log in pastebin.com already, and could you please take a look at that.

 Search for below topic.
How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

 

From the very beginning I have never bootstrap successfully, so I'm hoping someone from Cloudera can help me out with that.

 

Thanks.

Beatrice

Re: How to create a CDH-HA using the sample aws.ha.reference.conf provided by Cloudera

Explorer

Mike:

 

This has nothing to do with AWS access key or security group because I can create a AWS instance using the same security group and we have in our environment several EC2 servers with the security group defined.

 

Thanks.

Beatrice