Our security standards do not allow the OS type and version to be displayed when a user logons to the system.  That means our AMI images do not have the OS/Version listed in the /etc/issue file (since that file is displayed on logon).     Cloudera Director fails with the following error because the OS type is not listed in the/etc/issue file:   java.lang.UnsupportedOperationException: Operating system type not supported: UNKNOWN   The standard  RHEL 6.5 ami image's /etc/issue file contains the following (cat /etc/issue):   Red Hat Enterprise Linux Server release 6.5 (Santiago)     So I'm trying to determine a work around so that Cloudera Director can get the OS type another way since it is not listed in our image's /etc/issue file.   So my questions are:   When does Cloudera Director's script read the /etc/issue file?   Does it read the result of that file as it is displayed when connecting with SSH?   Or does it read the /etc/issue file after it connects?   Is there another option to skip the OS check or set something in our bootstrap script (that depends on when the file is read)?   Thanks!     ... View more

I am using the cloudera-director-client-1.1.3 to start up a small cluster.  I am having issues with Zookeeper.  I tried to using a single and 3 zookeeper nodes.  Sometimes the script runs, but most of the time it fails.    If I try to access the logs I get:   [Errno 2] No such file or directory: '/var/log/zookeeper/zookeeper-cmf-CD-ZOOKEEPER-bsejIGrE-SERVER-ip-10-0-1-41.ec2.internal.log'   If I look at the node the folder is empty.   Here is the error that I think is causing the issue: Command ZkInit with ID 56 failed. resultMessage=Command aborted because of exception: Command timed-out after 90 seconds   Is there a way to increase the timeout?  Do I need to turn off the diagnostic data collection (could that be causing an issue)?   Here is a snippet of the log file:   [2015-07-30 14:40:09] INFO  [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Creating cluster services [2015-07-30 14:40:09] INFO  [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Assigning roles to instances [2015-07-30 14:40:09] INFO  [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 11 roles for service CD-HDFS-sxHsiZxW [2015-07-30 14:40:09] INFO  [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 6 roles for service CD-YARN-GqpLWLyB [2015-07-30 14:40:10] INFO  [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 1 roles for service CD-ZOOKEEPER-bsejIGrE [2015-07-30 14:40:10] INFO  [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 5 roles for service CD-HIVE-PJiYBqej [2015-07-30 14:40:10] INFO  [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 1 roles for service CD-HUE-GatHLWtp [2015-07-30 14:40:10] INFO  [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 1 roles for service CD-OOZIE-nQaBSyxZ [2015-07-30 14:40:11] INFO  [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Automatically configuring services and roles [2015-07-30 14:40:11] INFO  [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Applying custom configurations of services [2015-07-30 14:40:12] INFO  [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Configuring Hive Metastore database .... [2015-07-30 14:40:25] INFO  [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Creating Hive Metastore Database [2015-07-30 14:40:25] INFO  [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << None{} [2015-07-30 14:40:28] INFO  [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/3 [53, Deployment{name='dev-3 Deployment', hostname='10.0.1.90', port=7180, username='admin', man ... [2015-07-30 14:40:28] INFO  [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateHiveDatabase with ID 53 completed successfully. Details: ApiCommand{id=53, name=CreateHiveDatabase, startTime=Thu Jul 30 14:40:19 EDT 2015, endTime=Thu Jul 30 14:40:25 EDT 2015, active=false, success=true, resultMessage=Created Hive Metastore Database., serviceRef=ApiServiceRef{peerName=null, clusterName=dev-3, serviceName=CD-HIVE-PJiYBqej}, roleRef=null, hostRef=null, parent=null} [2015-07-30 14:40:28] INFO  [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << None{} [2015-07-30 14:40:30] INFO  [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting on First Run command] [2015-07-30 14:40:31] INFO  [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Waiting on First Run command   I see an error about collecting diagnostic data&colon; .... 2015-07-30 14:43:20] INFO  [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Collecting and downloading diagnostic data [2015-07-30 14:43:21] ERROR [pipeline-thread-1] - c.c.l.b.ClouderaManagerLogRetriever: Got exception while collecting diagnostic data javax.ws.rs.ServiceUnavailableException: null   And then it attempts to do the "first run" of the install: [2015-07-30 14:43:21] WARN  [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Failed to collect diagnostic data [2015-07-30 14:43:21] ERROR [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Command First Run with ID 54 failed. Details: ApiCommand{id=54, name=First Run, startTime=Thu Jul 30 14:40:20 EDT 2015, endTime=Thu Jul 30 14:43:20 EDT 2015, active=false, success=false, resultMessage=Failed to perform First Run of services., serviceRef=null, roleRef=null, hostRef=null, parent=null} [2015-07-30 14:43:21] ERROR [pipeline-thread-1] - c.c.l.b.UnboundedWaitForApiCommand: Command ZkInit with ID 56 failed. Details: ApiCommand{id=56, name=ZkInit, startTime=Thu Jul 30 14:41:50 EDT 2015, endTime=Thu Jul 30 14:43:20 EDT 2015, active=false, success=false, resultMessage=Command aborted because of exception: Command timed-out after 90 seconds, serviceRef=ApiServiceRef{peerName=null, clusterName=dssh-dev-3, serviceName=CD-ZOOKEEPER-bsejIGrE}, roleRef=ApiRoleRef{clusterName=dssh-dev-3, serviceName=CD-ZOOKEEPER-bsejIGrE, roleName=CD-ZOOKEEPER-bsejIGrE-SERVER-95a522458bc9844f970bdffc8e1a5c6f}, hostRef=null, parent=null} [2015-07-30 14:43:21] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager 'First Run' command execution failed: Failed to perform First Run of services.   Here is the cluster description:   # Cluster description cluster {     products {       CDH: 5     }     parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.3.3/"]     services: [HDFS, YARN, ZOOKEEPER, HIVE, HUE, OOZIE]     configs {       ZOOKEEPER {         zookeeper_datadir_autocreate: true       }     }     masters-1 {       count: 1       instance: ${instances.c32x} {         tags {           group: master         }       }       roles {         HDFS: [NAMENODE, GATEWAY]       }     }     masters-2 {       count: 1       instance: ${instances.c32x} {         tags {           group: master         }       }       roles {         ZOOKEEPER: [SERVER]         HDFS: [SECONDARYNAMENODE, GATEWAY]         YARN: [RESOURCEMANAGER, JOBHISTORY]       }     }     workers {       count: 2       minCount: 2       instance: ${instances.d2x} {         tags {           group: worker         }       }       roles {         HDFS: [GATEWAY, DATANODE]         HIVE: [GATEWAY]         YARN: [NODEMANAGER, GATEWAY]       }     }     gateways {       count: 1       instance: ${instances.c32x} {         tags {           group: gateway         }       }       roles {         HIVE: [GATEWAY, HIVEMETASTORE, HIVESERVER2]         HDFS: [GATEWAY, BALANCER, HTTPFS]         HUE: [HUE_SERVER]         OOZIE: [OOZIE_SERVER]       }     } }   ... View more