Setup cloudera director, manager and clusters on AWS from already setup EC2

[#1]

Hi

I am in the stage to setup cloudera platform on a AWS. I have three main concerns:

1- Firstly, can I use cloudera director to set up cloudera manager and clusters on EC2s which alreay are set on AWS. or I need to intiate alll the EC2 for cluster from cloudera director.

2- Secondly, which approach is the best in regard to any comapatability issue. if the cloudera director is used to setup all theCDH and clusters on AWS, is it the ideal case or just initiate instances on AWS and used it to setup cloudera on top of that.

3- Finally, Can I install streamset as part of CDH services via cloudera director? or I should install it seperatly after the Cloudera platform is installed.

I really appricite you help.

[#2 · Accepted solution]

1. No, Cloudera Director can't point to existing instances, the EC2 instances must be provisioned by Director.

2. Based on my answer to (1), I don't think this question still applies. In general though, since Director manages provisioning and preparing the instances as well as installing Cloudera Manager and CDH, you'll likely run into much less compatibility issues using Director than if you were doing any of these steps manually.

3. Director can install Streamset via CSD. You can refer to this post to see an example using a bootstrap script. Note that as of Director 2.5 it's actually easier now to specify CSDs and doesn't require a bootstrap script, this is documented here.

[#3 · Accepted solution]

1. No, Cloudera Director can't point to existing instances, the EC2 instances must be provisioned by Director.

2. Based on my answer to (1), I don't think this question still applies. In general though, since Director manages provisioning and preparing the instances as well as installing Cloudera Manager and CDH, you'll likely run into much less compatibility issues using Director than if you were doing any of these steps manually.

3. Director can install Streamset via CSD. You can refer to this post to see an example using a bootstrap script. Note that as of Director 2.5 it's actually easier now to specify CSDs and doesn't require a bootstrap script, this is documented here.

[#4]

Thanks for the reply.

I started to cloudera director on EC2. but I just have the issue of not starting. When I run the sudo service cloudera-director-server start, I do not have any error, but I can not open cloudera director UI on the port. I checked the logs in /var/log/cloudera-director-server/cloud-init.log, I have the following error:

Error: JAVA_HOME is not set and Java could not be found |

I check the path using echo $JAVA_HOME and I have the folloiwngs:

/opt/jdk/jdk1.8.0_151

Is there any specfic location that I need to set the JAVA_HOME?

[#5]

Try to use a newer version of Java 1.8,
Do you have other Java installed as well (openjdk)?

[#6 · Accepted solution]

Thanks for the reply,

I resolved the issue by the followings:

 I  added it to /etc/defaults/cloudera-director-server  and also set it as default for all services by adding the line below to /etc/systemd/system.conf

DefaultEnvironment="JAVA_HOME=/opt/jdk/jdk1.8.0_151"

[#7]

Hi,

As I am trying to install streamset beside via cloudera director, I am trying to have all my nodes on Centos 7.3. But I could not see a parcel for centos in streamst repository. So if I point my config file to the following adrees, do you think I will have a problem?

"https://archives.streamsets.com/datacollector/latest/parcel/"

What do you suggest? as I would like to have all my machines with the same image, should I go with other options like RHEL 7? I really appriciate your response.

Regards

Armen