Cloudera Manager 'First Run' command execution failed: Failed to perform First Run of services

[#1]

Attempting to provision a new environment containing cluster of 3 Kafka and Zookeeper nodes using the cloudera-director bootstrap-remote CLI utility. The bootstrap process fails with the following stack trace:

http://pastebin.com/Qyg9bL96

However, when a new cluster is created from the director UI, the process succeeds.

Here is our configuration file:

name: Atlas-Test

provider {
  type: aws

  publishAccessKeys: true

  region: us-west-2

  subnetId: subnet-daf4c9ad

  securityGroupsIds: sg-feae6f99

  instanceNamePrefix: test

  associatePublicIpAddresses: false

}

ssh {
  username: ec2-user
  privateKey: /home/ec2-user/atlas.pem
}

instances {
  default {
    type: m4.large
    image: ami-414b7271

    tags {
      owner: ${?USER}
    }
  }
}

cloudera-manager {

  instance: ${instances.default} {
    tags {
      application: "Cloudera Manager 5"
    }
  }

  enableEnterpriseTrial: true

}

cluster {

  products {
    CDH: 5 
  }

  services: [ZOOKEEPER, KAFKA]

  master {
    count: 3

    instance: ${instances.default} {
      tags {
        group: master
      }
    }

    roles {
      ZOOKEEPER: [SERVER]
    }
  }


  gateway {
    count: 3

    instance: ${instances.default} {
      tags {
        group: kafka 
      }
    }

    roles {
      KAFKA: [KAFKA_BROKER]
    }
  }

}

[#2 · Accepted solution]

The underlying cause of the failure seems to be a failure to start Kafla:

[2016-02-01 19:10:58] ERROR [pipeline-thread-35] - c.c.l.b.UnboundedWaitForApiCommand: Command Start with ID 72 failed. Details: ApiCommand{id=72, name=Start, startTime=Mon Feb 01 19:10:10 EST 2016, endTime=Mon Feb 01 19:10:10 EST 2016, active=false, success=false, resultMessage=Failed to start service., serviceRef=ApiServiceRef{peerName=null, clusterName=Atlas-Test, serviceName=CD-KAFKA-MnViYSKk}, roleRef=null, hostRef=null, parent=null}

Cloudera Manager command execution history should have more details. 

I believe get this error because the Kafka parcel is not distributed (as you know Kafka is not included in the CDH parcel). Please add "KAFKA: 0.8.2" to the products {} block. You may also need to add reference to the precise parcel repositories you want to use.