Reply
New Contributor
Posts: 2
Registered: ‎06-11-2018

Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

Hi,

 

I  have a small cluster (4 nodes) where I tried to install Spark 2 following these instructions.

 

Everything went smoothly until I tried to redeploy the client configuration and got the following error message:

 

Execute command Deploy Client Configuration on service Spark 2
Deploy Client Configuration failed.       
    Generate and deploy client configuration.
    Completed only 0/4 steps. First failure: Client configuration (id=14) on host xxx.xxx.xxx.xx (id=1) exited with 1 and expected 0.

 

In more detail, I get this error message for all 4 hosts in my cluster:

+ ROOT_DIR_NAME=/etc/spark2
+ '[' '!' -e /etc/spark2 ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/taskcontroller.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/container-executor.cfg ']'
++ basename /etc/spark2/conf
+ LINK_BASENAME=conf
+ [[ -d conf ]]
+ '[' -n scripts/control.sh ']'
+ chmod u+x /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/scripts/control.sh
+ echo 'invoking optional deploy script scripts/control.sh'
+ SRC_PATH=/run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/spark2-conf
+ '[' '!' -d /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/spark2-conf ']'
+ pushd /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/spark2-conf
+ /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-3404885726574068161/scripts/control.sh client
++ log 'Running Spark2 CSD control script...'
+++ date
++ timestamp='Mon Jun 11 21:10:15 WEST 2018'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Running Spark2 CSD control script...'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Running Spark2 CSD control script...'
Mon Jun 11 21:10:15 WEST 2018: Running Spark2 CSD control script...
++ log 'Detected CDH_VERSION of [5]'
+++ date
++ timestamp='Mon Jun 11 21:10:15 WEST 2018'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Detected CDH_VERSION of [5]'
++ echo 'Mon Jun 11 21:10:15 WEST 2018: Detected CDH_VERSION of [5]'
Mon Jun 11 21:10:15 WEST 2018: Detected CDH_VERSION of [5]
++ PATH_TO_JAVA=/bin/java
+++ /bin/java -version
+++ awk -F '"' '/version/ {print $2}'
++ JAVA_VER=
++ JAVA_VER_SHORT=
++ [[ '' != \1\.\8 ]]
++ echo 'Java version 1.8 is required for Spark 2.3.'
++ exit 1

 

The problem is I do have Java version 1.8 installed. When I run 'java -version' I get the following output:
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

 

And my JAVA_HOME = /usr/lib/jvm/jdk1.8.0_162

Can anyone help me figure this out, please? I really need to install Spark 2.

Explorer
Posts: 14
Registered: ‎11-02-2017

Re: Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

New Contributor
Posts: 2
Registered: ‎06-11-2018

Re: Spark 2 - Deploy Client Configuration failed. Java version 1.8 is required

I have:

 

- CDS 2.3 Release 2 with CDH 5.14.0

- Cloudera Manager version 5.14.0

- Scala isn't installed (it's not mandatory, is it?)

- Python versions 2.7 and 3.6

- JDK 8

 

Could the problem I'm having be caused by conflicting Pyhton versions? Even though the error message I'm getting refers to a problem with the Java version?

Announcements