Cloudera Director 2.0 infinite loop Bootstrapping a CM 5.5 Instance when using my own AWS AMI

[#1]

Hi,

I have successfully created and deployed a Director 2.0 instance (on an AWS EC2 instance of type c4.xlarge with standard AWS CentOS x86_64 6.7 HVM Marketplace image ami-7b81ca41).

Director is working no problems....I point my browser at the following, login and can select "Add Cloudera Manager":

http://ec2-obfuscated.ap-southeast-2.compute.amazonaws.com:7189/wizards/addDeployment.html

I try and add a Cloudera Manager 5.5 instance using the standard screen (page 31 of the Cloudera Director 2.0 User Guide).

I keep the use the "Embedded DB" option (so I assume a PostGreSQL database will be created for specific CM schemas for the services).

The only difference is that I "Create New Instance Template" and in the "Image (AMI) ID" drop-down window, I chose and AMI of my own (ami-62062201....it's a CentOS 6.7 HVM AMI that has a few CentOS packages added to it).

My AWS AMI image also has the /root file system pre-created as a 100GB EBS volume (to work-around the CentOS 6.5 HVM related bug that doesn't allow re-sizing of the 8GB EBS /root file system (listed on page 14 of the Cloudera Director 2.0 User Guide).

When I now try to use Director to create and bootstrap an new Cloudera Manager 5.5.x instance, the attempt at bootstrapping sits in an infinite loop showing the following in the /var/log/cloudera-director-server/applications.log.

Are you able to provide information as to how to stop this ?

Is it possibly an IPv4 or IPv6 firewall issue ?   Blocking port 7180 from Director ?  Or to the CM instance ?

[2016-01-28 13:55:46] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:55:46] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:55:47] INFO [io-thread-2] - ssh:10.0.0.232: Error: No matching Packages to list
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: Package Arch Version Repository Size
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: Installing:
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: cloudera-manager-daemons x86_64 5.5.1-1.cm551.p0.8.el6 cloudera-manager 470 M
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: Transaction Summary
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: Install 1 Package(s)
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: Total download size: 470 M
[2016-01-28 13:55:48] INFO [io-thread-2] - ssh:10.0.0.232: Installed size: 655 M
[2016-01-28 13:57:45] INFO [io-thread-2] - ssh:10.0.0.232: Installed:
[2016-01-28 13:57:45] INFO [io-thread-2] - ssh:10.0.0.232: cloudera-manager-daemons.x86_64 0:5.5.1-1.cm551.p0.8.el6
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Installing cloudera-manager-server package (2/2)]
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.c.launchpad.pipeline.AbstractJob: Installing cloudera-manager-server package (2/2)
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> InstallPackages$InstallOrUpgradePackage/5 [YUM, cloudera-manager-server, PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), delegate= ...
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobUncheckedWithOutputLogging/3 [sudo yum -C info 'cloudera-manager-server', 10.0.0.232, SshCredentials{username='centos', hasPasswo ...
[2016-01-28 13:57:45] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:57:45] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Loaded plugins: fastestmirror, presto
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Available Packages
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Name : cloudera-manager-server
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Arch : x86_64
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Version : 5.5.1
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Release : 1.cm551.p0.8.el6
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Size : 8.2 k
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Repo : cloudera-manager
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Summary : The Cloudera Manager Server
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: URL : http://www.cloudera.com
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: License : Proprietary
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Description : The Cloudera Manager Server manages the services making up a
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: : Hadoop cluster.
[2016-01-28 13:57:46] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8631, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:57:46] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobFailFastWithOutputLogging/3 [sudo yum -C list installed 'cloudera-manager-server' 2>&1 > /dev/null && echo "Package cloudera-man ...
[2016-01-28 13:57:46] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:57:46] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:57:46] INFO [io-thread-2] - ssh:10.0.0.232: Error: No matching Packages to list
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Package Arch Version Repository Size
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Installing:
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: cloudera-manager-server x86_64 5.5.1-1.cm551.p0.8.el6 cloudera-manager 8.2 k
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Transaction Summary
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Install 1 Package(s)
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Total download size: 8.2 k
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Installed size: 11 k
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Installed:
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: cloudera-manager-server.x86_64 0:5.5.1-1.cm551.p0.8.el6
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> InstallJdbcDriverPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), delegate=null} Instance{virtualInstance ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> InstallKerberosPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), delegate=null} Instance{virtualInstance ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Setting up embedded PostgreSQL database for Cloudera Manager]
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.launchpad.pipeline.AbstractJob: Setting up embedded PostgreSQL database for Cloudera Manager
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetupEmbeddedPostgreSQLDatabase/2 [PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), delegate=null} Instance{virtualInstance ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8647, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> InstallPackages/4 [[cloudera-manager-server-db-2], PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), delegat ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> InstallPackages$PerformPackageManagerSpecificPreProcessing/3 [YUM, PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), delegate=null} Instance{virtualIns ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Installing cloudera-manager-server-db-2 package (1/1)]
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.launchpad.pipeline.AbstractJob: Installing cloudera-manager-server-db-2 package (1/1)
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> InstallPackages$InstallOrUpgradePackage/5 [YUM, cloudera-manager-server-db-2, PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), dele ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobUncheckedWithOutputLogging/3 [sudo yum -C info 'cloudera-manager-server-db-2', 10.0.0.232, SshCredentials{username='centos', hasP ...
[2016-01-28 13:57:47] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:57:47] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:57:47] INFO [io-thread-2] - ssh:10.0.0.232: Loaded plugins: fastestmirror, presto
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Available Packages
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Name : cloudera-manager-server-db-2
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Arch : x86_64
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Version : 5.5.1
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Release : 1.cm551.p0.8.el6
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Size : 9.9 k
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Repo : cloudera-manager
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Summary : Embedded database for the Cloudera Manager Server
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: URL : http://www.cloudera.com
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: License : Proprietary
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Description : This package configures an "embedded" PostgreSQL server,
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: : running as user cloudera-scm on port 7432. Installing
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: : this package short-circuits the database configuration
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: : otherwise necessary in /etc/cloudera-scm-server/db.properties.
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: : Requires PostgreSQL version 8.4 or higher.
[2016-01-28 13:57:48] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8665, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:57:48] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobFailFastWithOutputLogging/3 [sudo yum -C list installed 'cloudera-manager-server-db-2' 2>&1 > /dev/null && echo "Package clouder ...
[2016-01-28 13:57:48] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:57:48] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:57:48] INFO [io-thread-2] - ssh:10.0.0.232: Error: No matching Packages to list
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: Package Arch Version Repository Size
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: Installing:
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: cloudera-manager-server-db-2
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: x86_64 5.5.1-1.cm551.p0.8.el6 cloudera-manager 9.9 k
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: Installing for dependencies:
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: postgresql x86_64 8.4.20-4.el6_7 updates 2.6 M
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: postgresql-libs x86_64 8.4.20-4.el6_7 updates 202 k
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: postgresql-server x86_64 8.4.20-4.el6_7 updates 3.4 M
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: Transaction Summary
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: ================================================================================
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: Install 4 Package(s)
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: Total download size: 6.2 M
[2016-01-28 13:57:49] INFO [io-thread-2] - ssh:10.0.0.232: Installed size: 29 M
[2016-01-28 13:57:52] INFO [io-thread-2] - ssh:10.0.0.232: Installed:
[2016-01-28 13:57:52] INFO [io-thread-2] - ssh:10.0.0.232: cloudera-manager-server-db-2.x86_64 0:5.5.1-1.cm551.p0.8.el6
[2016-01-28 13:57:52] INFO [io-thread-2] - ssh:10.0.0.232: Dependency Installed:
[2016-01-28 13:57:52] INFO [io-thread-2] - ssh:10.0.0.232: postgresql.x86_64 0:8.4.20-4.el6_7
[2016-01-28 13:57:52] INFO [io-thread-2] - ssh:10.0.0.232: postgresql-libs.x86_64 0:8.4.20-4.el6_7
[2016-01-28 13:57:52] INFO [io-thread-2] - ssh:10.0.0.232: postgresql-server.x86_64 0:8.4.20-4.el6_7
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Starting embedded PostgreSQL database]
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.c.launchpad.pipeline.AbstractJob: Starting embedded PostgreSQL database
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobUncheckedWithOutputLogging/3 [sudo service postgresql stop, 10.0.0.232, SshCredentials{username='centos', hasPassword=false, hasP ...
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:57:52] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:57:52] INFO [io-thread-2] - ssh:10.0.0.232:[ OK ]g postgresql service:
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8666, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobUncheckedWithOutputLogging/3 [sudo service cloudera-scm-server-db start && sleep 10, 10.0.0.232, SshCredentials{username='centos' ...
[2016-01-28 13:57:52] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:57:52] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:57:53] INFO [io-thread-2] - ssh:10.0.0.232: The files belonging to this database system will be owned by user "cloudera-scm".
[2016-01-28 13:57:53] INFO [io-thread-2] - ssh:10.0.0.232: This user must also own the server process.
[2016-01-28 13:57:53] INFO [io-thread-2] - ssh:10.0.0.232: The database cluster will be initialized with locale en_US.UTF8.
[2016-01-28 13:57:53] INFO [io-thread-2] - ssh:10.0.0.232: The default text search configuration will be set to "english".
[2016-01-28 13:57:53] INFO [io-thread-2] - ssh:10.0.0.232: fixing permissions on existing directory /var/lib/cloudera-scm-server-db/data ... ok
[2016-01-28 13:57:53] INFO [io-thread-2] - ssh:10.0.0.232: creating subdirectories ... ok
[2016-01-28 13:57:55] INFO [io-thread-2] - ssh:10.0.0.232: selecting default max_connections ... 100
[2016-01-28 13:57:55] INFO [io-thread-2] - ssh:10.0.0.232: selecting default shared_buffers ... 32MB
[2016-01-28 13:57:55] INFO [io-thread-2] - ssh:10.0.0.232: creating configuration files ... ok
[2016-01-28 13:57:56] INFO [io-thread-2] - ssh:10.0.0.232: creating template1 database in /var/lib/cloudera-scm-server-db/data/base/1 ... ok
[2016-01-28 13:57:56] INFO [io-thread-2] - ssh:10.0.0.232: initializing pg_authid ... ok
[2016-01-28 13:57:56] INFO [io-thread-2] - ssh:10.0.0.232: setting password ... ok
[2016-01-28 13:57:57] INFO [io-thread-2] - ssh:10.0.0.232: initializing dependencies ... ok
[2016-01-28 13:57:57] INFO [io-thread-2] - ssh:10.0.0.232: creating system views ... ok
[2016-01-28 13:57:57] INFO [io-thread-2] - ssh:10.0.0.232: loading system objects' descriptions ... ok
[2016-01-28 13:57:57] INFO [io-thread-2] - ssh:10.0.0.232: creating conversions ... ok
[2016-01-28 13:57:58] INFO [io-thread-2] - ssh:10.0.0.232: creating dictionaries ... ok
[2016-01-28 13:57:58] INFO [io-thread-2] - ssh:10.0.0.232: setting privileges on built-in objects ... ok
[2016-01-28 13:57:59] INFO [io-thread-2] - ssh:10.0.0.232: creating information schema ... ok
[2016-01-28 13:57:59] INFO [io-thread-2] - ssh:10.0.0.232: vacuuming database template1 ... ok
[2016-01-28 13:57:59] INFO [io-thread-2] - ssh:10.0.0.232: copying template1 to template0 ... ok
[2016-01-28 13:58:00] INFO [io-thread-2] - ssh:10.0.0.232: copying template1 to postgres ... ok
[2016-01-28 13:58:00] INFO [io-thread-2] - ssh:10.0.0.232: Success. You can now start the database server using:
[2016-01-28 13:58:00] INFO [io-thread-2] - ssh:10.0.0.232: postgres -D /var/lib/cloudera-scm-server-db/data
[2016-01-28 13:58:00] INFO [io-thread-2] - ssh:10.0.0.232: or
[2016-01-28 13:58:00] INFO [io-thread-2] - ssh:10.0.0.232: pg_ctl -D /var/lib/cloudera-scm-server-db/data -l logfile start
[2016-01-28 13:58:00] INFO [io-thread-2] - ssh:10.0.0.232: Adding configs
[2016-01-28 13:58:00] INFO [io-thread-2] - ssh:10.0.0.232: Creating SCM configuration file: /etc/cloudera-scm-server/db.properties
[2016-01-28 13:58:01] INFO [io-thread-2] - ssh:10.0.0.232: waiting for server to start.... done
[2016-01-28 13:58:01] INFO [io-thread-2] - ssh:10.0.0.232: server started
[2016-01-28 13:58:01] INFO [io-thread-2] - ssh:10.0.0.232: CREATE ROLE
[2016-01-28 13:58:03] INFO [io-thread-2] - ssh:10.0.0.232: CREATE DATABASE
[2016-01-28 13:58:03] INFO [io-thread-2] - ssh:10.0.0.232: Created db properties file /etc/cloudera-scm-server/db.properties
[2016-01-28 13:58:03] INFO [io-thread-2] - ssh:10.0.0.232: Creating DB amon for role ACTIVITYMONITOR
[2016-01-28 13:58:03] INFO [io-thread-2] - ssh:10.0.0.232: CREATE ROLE
[2016-01-28 13:58:05] INFO [io-thread-2] - ssh:10.0.0.232: CREATE DATABASE
[2016-01-28 13:58:05] INFO [io-thread-2] - ssh:10.0.0.232: Created DB for role ACTIVITYMONITOR
[2016-01-28 13:58:05] INFO [io-thread-2] - ssh:10.0.0.232: Creating DB rman for role REPORTSMANAGER
[2016-01-28 13:58:05] INFO [io-thread-2] - ssh:10.0.0.232: CREATE ROLE
[2016-01-28 13:58:07] INFO [io-thread-2] - ssh:10.0.0.232: CREATE DATABASE
[2016-01-28 13:58:07] INFO [io-thread-2] - ssh:10.0.0.232: Created DB for role REPORTSMANAGER
[2016-01-28 13:58:07] INFO [io-thread-2] - ssh:10.0.0.232: Creating DB nav for role NAVIGATOR
[2016-01-28 13:58:07] INFO [io-thread-2] - ssh:10.0.0.232: CREATE ROLE
[2016-01-28 13:58:08] INFO [io-thread-2] - ssh:10.0.0.232: CREATE DATABASE
[2016-01-28 13:58:08] INFO [io-thread-2] - ssh:10.0.0.232: Created DB for role NAVIGATOR
[2016-01-28 13:58:08] INFO [io-thread-2] - ssh:10.0.0.232: Creating DB navms for role NAVIGATORMETASERVER
[2016-01-28 13:58:08] INFO [io-thread-2] - ssh:10.0.0.232: CREATE ROLE
[2016-01-28 13:58:11] INFO [io-thread-2] - ssh:10.0.0.232: CREATE DATABASE
[2016-01-28 13:58:11] INFO [io-thread-2] - ssh:10.0.0.232: Created DB for role NAVIGATORMETASERVER
[2016-01-28 13:58:11] INFO [io-thread-2] - ssh:10.0.0.232: Enabled remote connections
[2016-01-28 13:58:12] INFO [io-thread-2] - ssh:10.0.0.232: waiting for server to shut down.... done
[2016-01-28 13:58:12] INFO [io-thread-2] - ssh:10.0.0.232: server stopped
[2016-01-28 13:58:12] INFO [io-thread-2] - ssh:10.0.0.232: DB initialization done.
[2016-01-28 13:58:13] INFO [io-thread-2] - ssh:10.0.0.232: waiting for server to start.... done
[2016-01-28 13:58:13] INFO [io-thread-2] - ssh:10.0.0.232: server started
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8667, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobFailFastWithUnboundedOutputCollector/3 [sudo cat /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.mgmt.properties, 10.0.0 ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:58:23] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # Auto-generated by initialize_embedded_db.sh
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: #
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # 20160128-135752
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: #
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # These are database settings for CM Manager
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: #
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.db.type=postgresql
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.db.host=localhost:7432
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.db.name=scm
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.db.user=scm
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.db.password=UbpxrP0x3O
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # Auto-generated by initialize_embedded_db.sh
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: #
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # 20160128-135752
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: #
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # These are database credentials for databases
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # created by "cloudera-scm-server-db" for
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # Cloudera Manager Management Services,
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # to be used during the installation wizard if
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # the embedded database route is taken.
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: #
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # The source of truth for these settings
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # is the Cloudera Manager databases and
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # changes made here will not be reflected
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: # there automatically.
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: #
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.ACTIVITYMONITOR.db.type=postgresql
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.ACTIVITYMONITOR.db.host=ip-10-0-0-232.ap-southeast-2.compute.internal:7432
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.ACTIVITYMONITOR.db.name=amon
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.ACTIVITYMONITOR.db.user=amon
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.ACTIVITYMONITOR.db.password=
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.REPORTSMANAGER.db.type=postgresql
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.REPORTSMANAGER.db.host=ip-10-0-0-232.ap-southeast-2.compute.internal:7432
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.REPORTSMANAGER.db.name=rman
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.REPORTSMANAGER.db.user=rman
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.REPORTSMANAGER.db.password=
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATOR.db.type=postgresql
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATOR.db.host=obfuscated.ap-southeast-2.compute.internal:7432
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATOR.db.name=nav
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATOR.db.user=nav
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATOR.db.password=
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATORMETASERVER.db.type=postgresql
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATORMETASERVER.db.host=obfuscated.ap-southeast-2.compute.internal:7432
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATORMETASERVER.db.name=navms
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATORMETASERVER.db.user=navms
[2016-01-28 13:58:23] INFO [io-thread-2] - 10.0.0.232: com.cloudera.cmf.NAVIGATORMETASERVER.db.password=
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8668, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetupEmbeddedPostgreSQLDatabase$ParseDatabaseProperties/1 [[# Auto-generated by initialize_embedded_db.sh
, #
, # 20160128-135752
, #
, # These are databa ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8669, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManager$MergeDatabaseConfigurations/2 [{CLOUDERA_MANAGER=ExternalDatabase{type=POSTGRESQL, host='localhost', port=7432, name='scm', user=' ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8670, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Starting Cloudera Manager server]
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.launchpad.pipeline.AbstractJob: Starting Cloudera Manager server
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SshJobUncheckedWithOutputLogging/3 [sudo service cloudera-scm-server start, 10.0.0.232, SshCredentials{username='centos', hasPassword=f ...
[2016-01-28 13:58:23] INFO [pipeline-thread-4] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-01-28 13:58:23] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.232 was automatically accepted
[2016-01-28 13:58:28] INFO [io-thread-2] - ssh:10.0.0.232:[FAILED]g cloudera-scm-server:
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8671, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for Cloudera Manager server to start]
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager server to start
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> WaitForInstanceOnPortUntilTime/4 [PluggableComputeInstance{ipAddress=Optional.of(10.0.0.232), delegate=null} Instance{virtualInstance ...
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=8678, pipeline=2d6acf22-d593-41ee-90e7-3103138254a6, ...
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.pipeline.util.PipelineRunner: >> WaitForServerOnPortUntilTime/4 [10.0.0.232, 7180, Optional.absent(), Optional.absent()]
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Unbounded wait for server 10.0.0.232 to listen on port 7180
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Checking every 10000 milliseconds
[2016-01-28 13:58:28] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180
[2016-01-28 13:58:38] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180
[2016-01-28 13:58:48] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180
[2016-01-28 13:58:58] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180

.....

.....

.....

[2016-01-28 14:16:29] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180
[2016-01-28 14:16:39] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180
[2016-01-28 14:16:49] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180
[2016-01-28 14:16:59] INFO [pipeline-thread-4] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.0.232:7180

[#2]

Hello,

Thank you for providing the detailed steps and logs. Based on the logs, Director was able to set up CM and the Management services on the embedded db, but is stuck waiting for CM to start.

As a starting point, could you check on the CM host (10.0.0.232) if it is up and running? If you can ssh into this,

a) ps -ef | grep cloudera-scm-server => This will show you if the process is up and running

b) netstat -ntulp | grep 7180 => This will show you what process id is on port 7180, and in the ideal state it would be the same as that from 'a'

If the server is down, it's possible that the system did not have sufficient capacity. (dmesg may show a process killed because of OOM).

What instance type did you use for the CM server? We recommend m4.xlarge or higher. Detailed resources requirements can be found in http://www.cloudera.com/documentation/director/latest/PDF/cloudera-director.pdf (pg 18, 19)

If the CM process is up and listening on 7180, could you check if the CM server log has any errors? You can find it at /var/log/cloudera-scm-server/cloudera-scm-server.log

Let me know what you find, and we can take it from there.

Thanks,

Jayita

[#3]

Hello,

If the CM server is up and running, it's possible that the security group used by the CM instance does not accept connections on any port other than ssh. Could you verify this, and if the change is needed, modify the current security group and retry?

There is some information and recommendations here: https://www.cloudera.com/documentation/director/latest/topics/director_deployment_start_launcher.htm...

- Jayita

[#4]

Hi Jayita,

Thankyou for your quick response !

If I can address each of your responses as follows....

a)  I can successfully ssh into the pending CM host 10.0.0.232 (the AWS EC2 instance that Director has started to bootstrap).

b)  When I run "ps -afe | grep cloudera-scm-sever", I can see the process running.

c)  I've tried "netstat -al" | less.....but cannot see an entry showing the CM port 7180 on the pending CM host 10.0.0.232 as being open (or any mention of 7180 at all).

My educated guess is that the ideal state would be something like the following (but this is not shown):

[root@ip-obfuscated]# netstat -al | grep 7180
tcp 0 0 *:7180 *:* LISTEN
tcp 0 0 ip-obfuscated:7180   ESTABLISHED

The CM instance is an AWS EC2 instance of type "m4.xlarge".....configured with 100GB /root EBS volume (so plenty of "grunt").

Both the Director instance and the pending bootstraping CM instance are located in the same AWS VPC, same Subnet, same Security Group with the _exact_ inbound rules as shown on page 22 of the Cloudera Director 2.0 Users Guide.

Late yesterday afternoon I terminated the bootstrapping CM instance....

I will again today to create a new CM instance via Director (with an AWS AMI image that is not the one I created.....I will use AWS AMI ami-7b81ca41 (the standard CentOS 6.7 x86_64 HVM instance).

I will give you an update....

Thanks,

Damion.

[#5]

Hi Damion,

Did trying with a different AMI resolve the issue for you?

- Jayita

[#6]

Hi Jayita,

Yes, I've started using RHEL 7.1 AMI images from the standard AWS Marketplace (rather than my own AMI made from a CentOS 6.7 base and rather than the AWS Marketplace CentOS 6.4 AMI).

I think the issue was in my own CentOS 6.7 AMI.  I had disabled the IPv6 firewall and somehow blocked port 7180 on the CM 5.5.x instance, which in turn prevented the Director instance from accessing it via port 7180 during installation.

I've gone back to the RHEL 7.1 AWS AMI for this reason (and for another reason....using CentOS AMI's still prevents the /root file systme from being re-sized above the standard 8GB during install by Director (not a Director issue, but a CentOS bug I believe).

Installation of Director 2.0.0 (on an EC2 instance using RHEL 7.1 AMI) is very simple.

Installation of CM 5.5.x (on an EC2 instance using RHEL 7.1 AMI) and creating a "Core Hadoop" or any other optional type of CDH cluster (like with the Hive, Impala and/or Spark services) using Director 2.0.0 is almost flawless.

The only comments I'd like to make relate to the following:

1) Timezone setup when creating the Director 2.0.0, CM 5.5.x and Master/Worker/Gateway nodes....

Even after Director 2.0.0, CM 5.5.x and each node in the cluster is installed, there is still a manual step that must be performed on each node....and that's changing each systems timezone to reflect my timezone (using the RHEL 7.1 "timedatectl" utility).

It would be nice if Director 2.0.x could feature a screen prompt during installation asking for the timezone and updating this (if not, no big deal, I can simply add it as a post-bootstrap script).

2) Using external databases to replace the Director "H2" and CM databases....

It's quite a bit confusing in the Director 2.0.0 User Guide on how to do this.  Better explanation on how to do it during the install would be fantastic.  

Perhaps there's a Cloudera Knowledge Base Article somewhere ?

3) Director Server Templates....

Is there a way I can "edit" (or remove) some of the Director templates ?   When I go into the Director URL and into each "Environment" I can see all the templates I've created.

Is there a way I can delete some of them ?

Is there a way I can "export" a template to an JSON/XML file ?

Is there a way I can "copy" templates between different Director environments (ie"  from out On-Demand-Dev environment to our On-Demand-Test environment so we can create the same instances/nodes without "cloning" an entire cluster) ?

Regards,

Damion.

[#7 · Accepted solution]

Hi Damion,

Thanks for providing the details on how you resolved the access issues to port 7180. 

Regarding your other points,

#1 - Thanks for the feedback. This will be evaluated and prioritized, but in the meanwhile please make this change within your bootstrap script. 

#2 - I will pass this feedback on to our Documentation team. Could you share some more details on which parts of the documentation were confusing or unclear - was it how to set up MySQL server, or how to set up the Director database on it? To clarify, this has to be done for Director when it is first set up (and not a migration from an existing H2 database to MySQL).

Regarding external databases for CM and CDH, you could use Director to create the external DB for you, or create one yourself and have Director use it for CM and CDH. If there was something specific about this documentation too that was unclear, please let me know. 

Some documentation references:

For Director: https://www.cloudera.com/documentation/director/latest/topics/director_use_ext_db_for_director_data....

For CM: https://www.cloudera.com/documentation/director/latest/topics/director_external_db.html

#3 Instance templates:

- Yes you can edit/delete what you see under the Environment. Note that this does not affect existing clusters or instances.

- Director provides an API console. You can find a link to this under the help (question mark) icon in the top right navigation menu. In the console, there is a section on Instance Templates, and you can use this to retrieve the template contents in JSON.

- We don't provide the copy as you describe it. You could use the above console to GET from one environment and PUT in another, i.e. essentially create new templates based on the existing ones. This only creates the template though, it doesn't automatically create instances (nodes) or clusters. Those would have to be re-created in the second environment using the new templates.

[#8]

I'm facing the same issue on AWS with CentOS. I just don't agree to change my AMI to RHEL as it would cost more. My problem happens every time I start an instance using Director(with an CentOS AMI), it gets in a loop until time out. The diference is that I can not acess the instance using SSH. RHEL works fine.

Do you guys have any reference on it? 

[#9]

When you say you're having the same issue, I need some clarification. The original poster was complaining that Director could not make an HTTP connection to Cloudera Manager on port 7180. You seem to be saying that Director (and you yourself) cannot establish an SSH connection to an instance (on port 22). These are likely two different problems. If this is correct, can you please open a separate question, and post a snippet from Director's application.log showing the failure to establish a connection? That way future forum users can benefit from whatever solution we find.