Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark 2 Installation Failed

avatar
Explorer

Hello,

 

I tried to install Spark 2 by using CSD on cloudera manager but I got error while deploying client configuration. 

 

I followed all the steps in the link below; 

 

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

 

My CDH version is 5.14 but there is no spark2 parcel for 5.14, there are parcels for 5.13 and 5.12. Is this the problem I faced with?

 

Could you help me on this?

 

Spark2.PNG

BR,

 

Huriye 

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Ok, from the log it is obvioues that issue for spark is the old jdk.

 

 

When you tried to upgrade java have you defined the java home in "/etc/default/cloudera-scm-server"

e.g.:

export JAVA_HOME="/usr/lib/jvm/java-8-oracle/"

Can you send the relevant "/var/log/cloudera-scm-server/cloudera-scm-server.out" ?

 

View solution in original post

5 REPLIES 5

avatar
Super Collaborator
My CDH version is 5.14 but there is no spark2 parcel for 5.14, there are parcels for 5.13 and 5.12. Is this the problem I faced with?

This is not an issue. Spark is built on CDH5.13 but it works fine with CDH5.14.

Check compatibility notes:

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html#cdh_versions

 

According to the screenshot, the procedure failed while it was distributing configuration to namenode. Can you check the "stdout" and "stderr" output? You can copy it here so we can take a look.

avatar
Explorer

Hello,

 

stdout and stderr are in below;

 

Tue Jun 26 09:01:24 +03 2018
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771 as CONF_DIR
using spark2-conf as DIRECTORY_NAME
using /etc/spark2/conf.cloudera.spark2_on_yarn as DEST_PATH
using spark2-conf as ALT_NAME
using /etc/spark2/conf as ALT_LINK
using 51 as PRIORITY
using scripts/control.sh as RUNNER_PROGRAM
using client as RUNNER_ARGS
using /usr/sbin/update-alternatives as UPDATE_ALTERNATIVES
Deploying service client configs to /etc/spark2/conf.cloudera.spark2_on_yarn
invoking optional deploy script scripts/control.sh
/run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771
Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...
Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]
Java version 1.8 is required for Spark 2.2.
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/taskcontroller.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/container-executor.cfg ']'
++ basename /etc/spark2/conf
+ LINK_BASENAME=conf
+ [[ -d conf ]]
+ '[' -n scripts/control.sh ']'
+ chmod u+x /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/scripts/control.sh
+ echo 'invoking optional deploy script scripts/control.sh'
+ SRC_PATH=/run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf
+ '[' '!' -d /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf ']'
+ pushd /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf
+ /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/scripts/control.sh client
++ log 'Running Spark2 CSD control script...'
+++ date
++ timestamp='Tue Jun 26 09:01:24 +03 2018'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...'
Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...
++ log 'Detected CDH_VERSION of [5]'
+++ date
++ timestamp='Tue Jun 26 09:01:24 +03 2018'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]'
Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]
++ PATH_TO_JAVA=/usr/java/jdk1.7.0_67-cloudera/bin/java
+++ /usr/java/jdk1.7.0_67-cloudera/bin/java -version
+++ awk -F '"' '/version/ {print $2}'
++ JAVA_VER=1.7.0_67
++ JAVA_VER_SHORT=1.7
++ [[ 1.7 != \1\.\8 ]]
++ echo 'Java version 1.8 is required for Spark 2.2.'
++ exit 1

 

 

avatar
Expert Contributor

Hi @Huriye

 

Thank you for posting your query with Cloudera community.

 

From the documentation I could see, the requirement for spark 2.2.x is JDK 1.8

 

Please refer the below URL

 

[1]. https://www.cloudera.com/documentation/spark2/2-2-x/topics/spark2_requirements.html

Thanks,
Satz

avatar
Explorer

Hello,

 

Thank you for your answer, I tried to change java version but cloudera service not starting.

 

I followed th link: https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cdh_cm_upgrading_to_jdk8.html

 

But I get error which is Failed to start LSB: Cloudera SCM Server and there is no error on /var/log/cloudera-scm-server

 

Best regards,

 

Huriye

avatar
Super Collaborator

Ok, from the log it is obvioues that issue for spark is the old jdk.

 

 

When you tried to upgrade java have you defined the java home in "/etc/default/cloudera-scm-server"

e.g.:

export JAVA_HOME="/usr/lib/jvm/java-8-oracle/"

Can you send the relevant "/var/log/cloudera-scm-server/cloudera-scm-server.out" ?