Java 8 on CDH cluster using Cloudera director (AWS)

[#1]

We require java 8 on the CDH cluster to support analytics tools so Cloudera’s suggestion to how to solve this would be very helpful at this point. We have a CDH cluster on AWS that was spun up using Cloudera Director. Unfortunately, that spun up with java 1.6 and 1.7 as Cloudera director automatically installs and points to 1.6

The upgrade to 1.8.0_60 isn’t going so smooth. We are going to put that aside for the moment and look to get a cluster spun up with 1.8.0_60 from the start. Is there a suggested path from Cloudera on this? We want to use Cloudera Director to handle the installs and looking for an official Cloudera response as to the use of java 8 as ‘the’ java to use for the cluster.

Cloudera’s docs on java 8 upgrade are abstracted out to a high level so obviously there are issues to get past. The issue I am seeing currently is on a data node and trying to spin it up after updating to 1.8.0_60. Screenshot attached.

 

[#2 · Accepted solution]

sunilcsc,

It is possible to create a cluster with java 8 through Director.

The Deployment (CM) must be created with javaInstallationStrategy set to NONE. This setting tells CM not to install java on the CDH hosts. This property is not exposed through the UI so you need to create the cluster using a conf file.

...
cloudera-manager {
   ...
   javaInstallationStrategy: NONE
   ...
}
...

Java 8 needs to be installed on each instance using a bootstrap script.

Here is an example bootstrap script you can use as a starting point. It removes openjdk, downloads a java 8 rpm, and installs it.

#!/bin/sh

yum remove --assumeyes *openjdk*
yum install --assumeyes wget
cd /opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.rpm"
yum install --assumeyes /opt/jdk-8u45-linux-x64.rpm

[#3 · Accepted solution]

sunilcsc,

It is possible to create a cluster with java 8 through Director.

The Deployment (CM) must be created with javaInstallationStrategy set to NONE. This setting tells CM not to install java on the CDH hosts. This property is not exposed through the UI so you need to create the cluster using a conf file.

...
cloudera-manager {
   ...
   javaInstallationStrategy: NONE
   ...
}
...

Java 8 needs to be installed on each instance using a bootstrap script.

Here is an example bootstrap script you can use as a starting point. It removes openjdk, downloads a java 8 rpm, and installs it.

#!/bin/sh

yum remove --assumeyes *openjdk*
yum install --assumeyes wget
cd /opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.rpm"
yum install --assumeyes /opt/jdk-8u45-linux-x64.rpm

[#4 · Accepted solution]

It should be noted that the structure of this script works, but I'd advise you to use a more stable location for that RPM than Oracle's site. Oracle only allows you to download the 2 most recent versions of Java 8 without logging in, which this script does not do. Java 8u45 likely requires this login to download. You could use something like an internal S3 bucket (provided you're on AWS) to host it and download it internally.

[#5]

Perfect. Exactly what I was looking for. Thank you!

I will take and put java in an S3 bucket as suggested below as well.

Many thanks!

[#6]

When will Director support this natively?

[#7]

Hi, we currently don't have a scheduled date for this, but we will take
your request into consideration during prioritization.

[#8]

I wanted to start a new thread but having very similiar question. Why is Cloudera Manager / Director deploying java 1.6 when Cloudera clearly states that the lowest possible supported version is 1.7?

Thanks for the clarification

https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_ig_req_supported_versions.html#co...

[#9]

Director installs Java 7 by default and has since version 1.0. Is it possible that Java is unexpectedly installed on the instance type that you're using? Director attempts to remove existing Java versions prior to installing its own version, but occasionally certain images have Java installed in ways that Director isn't anticipating.

What version of Director are you using? Assuming that you're using AWS: what region and what AMI are you using to try to bootstrap the cluster?

[#10]

The latest version of Director (2.4 I assume).
I really tooked a simple wizard, went throught with 1 master, 3 workers, 1 gateway, installed the cluster and checked the java in every host (1.6)
AMI Centos 7, tried region eu-west-1, us-east-1.

[#11]

Can you give me the exact AMI IDs that you are using to try out?