Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Java 8 on CDH cluster using Cloudera director (AWS)

Solved Go to solution

Java 8 on CDH cluster using Cloudera director (AWS)

New Contributor

We require java 8 on the CDH cluster to support analytics tools so Cloudera’s suggestion to how to solve this would be very helpful at this point. We have a CDH cluster on AWS that was spun up using Cloudera Director. Unfortunately, that spun up with java 1.6 and 1.7 as Cloudera director automatically installs and points to 1.6

 

The upgrade to 1.8.0_60 isn’t going so smooth. We are going to put that aside for the moment and look to get a cluster spun up with 1.8.0_60 from the start. Is there a suggested path from Cloudera on this? We want to use Cloudera Director to handle the installs and looking for an official Cloudera response as to the use of java 8 as ‘the’ java to use for the cluster.

 

Cloudera’s docs on java 8 upgrade are abstracted out to a high level so obviously there are issues to get past. The issue I am seeing currently is on a data node and trying to spin it up after updating to 1.8.0_60. Screenshot attached.

 Screen Shot 2016-07-06 at 4.53.44 PM.png

 

 

2 ACCEPTED SOLUTIONS

Accepted Solutions

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Expert Contributor

sunilcsc,

 

It is possible to create a cluster with java 8 through Director.

 

The Deployment (CM) must be created with javaInstallationStrategy set to NONE. This setting tells CM not to install java on the CDH hosts. This property is not exposed through the UI so you need to create the cluster using a conf file.

...
cloudera-manager {
   ...
   javaInstallationStrategy: NONE
   ...
}
...

 

Java 8 needs to be installed on each instance using a bootstrap script.

Here is an example bootstrap script you can use as a starting point. It removes openjdk, downloads a java 8 rpm, and installs it.

 

#!/bin/sh

yum remove --assumeyes *openjdk*
yum install --assumeyes wget
cd /opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.rpm"
yum install --assumeyes /opt/jdk-8u45-linux-x64.rpm

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Expert Contributor

It should be noted that the structure of this script works, but I'd advise you to use a more stable location for that RPM than Oracle's site. Oracle only allows you to download the 2 most recent versions of Java 8 without logging in, which this script does not do. Java 8u45 likely requires this login to download. You could use something like an internal S3 bucket (provided you're on AWS) to host it and download it internally.

9 REPLIES 9

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Expert Contributor

sunilcsc,

 

It is possible to create a cluster with java 8 through Director.

 

The Deployment (CM) must be created with javaInstallationStrategy set to NONE. This setting tells CM not to install java on the CDH hosts. This property is not exposed through the UI so you need to create the cluster using a conf file.

...
cloudera-manager {
   ...
   javaInstallationStrategy: NONE
   ...
}
...

 

Java 8 needs to be installed on each instance using a bootstrap script.

Here is an example bootstrap script you can use as a starting point. It removes openjdk, downloads a java 8 rpm, and installs it.

 

#!/bin/sh

yum remove --assumeyes *openjdk*
yum install --assumeyes wget
cd /opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.rpm"
yum install --assumeyes /opt/jdk-8u45-linux-x64.rpm

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Expert Contributor

It should be noted that the structure of this script works, but I'd advise you to use a more stable location for that RPM than Oracle's site. Oracle only allows you to download the 2 most recent versions of Java 8 without logging in, which this script does not do. Java 8u45 likely requires this login to download. You could use something like an internal S3 bucket (provided you're on AWS) to host it and download it internally.

Re: Java 8 on CDH cluster using Cloudera director (AWS)

New Contributor

Perfect. Exactly what I was looking for. Thank you!

 

I will take and put java in an S3 bucket as suggested below as well.

 

Many thanks!

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Contributor

When will Director support this natively?

Highlighted

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Contributor
Hi, we currently don't have a scheduled date for this, but we will take
your request into consideration during prioritization.

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Master Collaborator

I wanted to start a new thread but having very similiar question. Why is Cloudera Manager / Director deploying java 1.6 when Cloudera clearly states that the lowest possible supported version is 1.7?

 

Thanks for the clarification

 

https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_ig_req_supported_versions.html#co...

 

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Expert Contributor
Director installs Java 7 by default and has since version 1.0. Is it possible that Java is unexpectedly installed on the instance type that you're using? Director attempts to remove existing Java versions prior to installing its own version, but occasionally certain images have Java installed in ways that Director isn't anticipating.

What version of Director are you using? Assuming that you're using AWS: what region and what AMI are you using to try to bootstrap the cluster?

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Master Collaborator
The latest version of Director (2.4 I assume).
I really tooked a simple wizard, went throught with 1 master, 3 workers, 1 gateway, installed the cluster and checked the java in every host (1.6)
AMI Centos 7, tried region eu-west-1, us-east-1.

Re: Java 8 on CDH cluster using Cloudera director (AWS)

Expert Contributor
Can you give me the exact AMI IDs that you are using to try out?
Don't have an account?
Coming from Hortonworks? Activate your account here