Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

New Contributor

Hi,

 

I  have a small cluster (4 nodes) where I tried to install Spark 2 following these instructions.

 

Everything went smoothly until I tried to redeploy the client configuration and got the following error message:

 

Execute command Deploy Client Configuration on service Spark 2
Deploy Client Configuration failed.       
    Generate and deploy client configuration.
    Completed only 0/4 steps. First failure: Client configuration (id=14) on host xxx.xxx.xxx.xx (id=1) exited with 1 and expected 0.

 

In more detail, I get this error message for all 4 hosts in my cluster:

+ ROOT_DIR_NAME=/etc/spark2
+ '[' '!' -e /etc/spark2 ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/taskcontroller.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/container-executor.cfg ']'
++ basename /etc/spark2/conf
+ LINK_BASENAME=conf
+ [[ -d conf ]]
+ '[' -n scripts/control.sh ']'
+ chmod u+x /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/scripts/control.sh
+ echo 'invoking optional deploy script scripts/control.sh'
+ SRC_PATH=/run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/spark2-conf
+ '[' '!' -d /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/spark2-conf ']'
+ pushd /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/spark2-conf
+ /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/scripts/control.sh client
++ log 'Running Spark2 CSD control script...'
+++ date
++ timestamp='Mon Jun 11 21:10:15 WEST 2018'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Running Spark2 CSD control script...'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Running Spark2 CSD control script...'
Mon Jun 11 21:10:15 WEST 2018: Running Spark2 CSD control script...
++ log 'Detected CDH_VERSION of [5]'
+++ date
++ timestamp='Mon Jun 11 21:10:15 WEST 2018'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Detected CDH_VERSION of [5]'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Detected CDH_VERSION of [5]'
Mon Jun 11 21:10:15 WEST 2018: Detected CDH_VERSION of [5]
++ PATH_TO_JAVA=/bin/java
+++ /bin/java -version
+++ awk -F '"' '/version/ {print $2}'
++ JAVA_VER=
++ JAVA_VER_SHORT=
++ [[ '' != \1\.\8 ]]
++ echo 'Java version 1.8 is required for Spark 2.3.'
++ exit 1

 

The problem is I do have Java version 1.8 installed. When I run 'java -version' I get the following output:
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

 

And my JAVA_HOME = /usr/lib/jvm/jdk1.8.0_162

Can anyone help me figure this out, please? I really need to install Spark 2.

3 REPLIES 3
Highlighted

Re: Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

Explorer

Re: Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

New Contributor

I have:

 

- CDS 2.3 Release 2 with CDH 5.14.0

- Cloudera Manager version 5.14.0

- Scala isn't installed (it's not mandatory, is it?)

- Python versions 2.7 and 3.6

- JDK 8

 

Could the problem I'm having be caused by conflicting Pyhton versions? Even though the error message I'm getting refers to a problem with the Java version?

Re: Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

New Contributor

you need to setting java-8-oracle path in cloudera UI.

 

cloudera homepage - host - configuration - Category > Advanced - Set the Java Home Directory property to the custom location

 

good luck!