Reply
Explorer
Posts: 17
Registered: ‎11-22-2017

roles for different service on cloudera

[ Edited ]

Hi

I am trying to provision the cloudera platform on AWS instances via cloudera director. I got the following concerns regarding this:

 

1- I will use m4 type of instance on AWS. These types of instance need the EBS storage for provisioning. I am wondering how I can allocated the EBS  and number of storage for instance. In the example config file, it is just the type as follow:

 

Spoiler
m4x : ${common-instance-properties} {
type: m4.xlarge
# image: ami-REPLACE-ME
}

2- I also wnat to define the role of different services on the nodes. Is there a comperhensive list that I can check for each service, what types of roles need to be defined for each service. for instance, I want to use KAFKA on gateway node and I used the following: Do I need to define any  more role for master or worker?

Spoiler
gateways-1 {
count: 2

instance: ${instances.m42x} {
tags {
group: gateway
}
}

roles {
# HBASE: [GATEWAY]
# HIVE: [GATEWAY]
KAFKA: [KAFKA_BROKER]
}

# Optional custom role configurations
# Configuration keys containing periods must be enclosed in double quotes.
# configs {
# }
}

3- How can I setup the hadoop, HIVE, Hue command lines. Are those the gateway roles whcih I can setup on gateway node like this:

Spoiler
roles {
HBASE: [GATEWAY]
HIVE: [GATEWAY]
HDFS: [GATEWAY]
}

 4- I noticed that cloud manager instance is provisioned seperatley. IS it possibe to put other services in this instance?

 

5- The last but not least, If I put some services and their roles, can I later use Cloudera manager to change the services and roles? In other words, is cloudera director is the intial setup and the final setup of services can be done on cloudera manager?

Highlighted
Cloudera Employee
Posts: 24
Registered: ‎08-24-2015

Re: roles for different service on cloudera

1. Yes you can place the EBS properties in that section of the conf file. See here on the available EBS properties.

 

2. The KAFKA_BROKER role should not be placed in a gateway group, there should be a Kafka GATEWAY role for this. The sample reference configuration file provided when Director is installed (can also find here) has a fairly comprehensive list of services and roles available. Although it looks like this doc needs to be updated since it's missing the gateway role for Kafka.

 

3. Yes, having these gateway roles should propagate the necessary configs and allow using the cli.

 

4. No, Director doesn't allow placing cluster services in the CM instance.

 

5. Yes, Director attempts to sync with Cloudera Manager every few minutes, so you should be abel to make direct changes in CM as long as the change is consistant accross the instance group. Refer here for additional details.

Announcements