Spark 2 Installation Failed

[#1]

Hello,

I tried to install Spark 2 by using CSD on cloudera manager but I got error while deploying client configuration. 

I followed all the steps in the link below; 

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

My CDH version is 5.14 but there is no spark2 parcel for 5.14, there are parcels for 5.13 and 5.12. Is this the problem I faced with?

Could you help me on this?

BR,

Huriye 

[#2]

My CDH version is 5.14 but there is no spark2 parcel for 5.14, there are parcels for 5.13 and 5.12. Is this the problem I faced with?

This is not an issue. Spark is built on CDH5.13 but it works fine with CDH5.14.

Check compatibility notes:

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html#cdh_versions

According to the screenshot, the procedure failed while it was distributing configuration to namenode. Can you check the "stdout" and "stderr" output? You can copy it here so we can take a look.

[#3]

Hello,

stdout and stderr are in below;

Tue Jun 26 09:01:24 +03 2018
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771 as CONF_DIR
using spark2-conf as DIRECTORY_NAME
using /etc/spark2/conf.cloudera.spark2_on_yarn as DEST_PATH
using spark2-conf as ALT_NAME
using /etc/spark2/conf as ALT_LINK
using 51 as PRIORITY
using scripts/control.sh as RUNNER_PROGRAM
using client as RUNNER_ARGS
using /usr/sbin/update-alternatives as UPDATE_ALTERNATIVES
Deploying service client configs to /etc/spark2/conf.cloudera.spark2_on_yarn
invoking optional deploy script scripts/control.sh
/run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771
Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...
Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]
Java version 1.8 is required for Spark 2.2.

+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/taskcontroller.cfg ']'
+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg}'
+ '[' -e /etc/spark2/conf.cloudera.spark2_on_yarn/container-executor.cfg ']'
++ basename /etc/spark2/conf
+ LINK_BASENAME=conf
+ [[ -d conf ]]
+ '[' -n scripts/control.sh ']'
+ chmod u+x /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/scripts/control.sh
+ echo 'invoking optional deploy script scripts/control.sh'
+ SRC_PATH=/run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf
+ '[' '!' -d /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf ']'
+ pushd /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/spark2-conf
+ /run/cloudera-scm-agent/process/ccdeploy_spark2-conf_etcspark2conf.cloudera.spark2_on_yarn_-1963022203308540771/scripts/control.sh client
++ log 'Running Spark2 CSD control script...'
+++ date
++ timestamp='Tue Jun 26 09:01:24 +03 2018'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...'
Tue Jun 26 09:01:24 +03 2018: Running Spark2 CSD control script...
++ log 'Detected CDH_VERSION of [5]'
+++ date
++ timestamp='Tue Jun 26 09:01:24 +03 2018'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]'
++ echo 'Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]'
Tue Jun 26 09:01:24 +03 2018: Detected CDH_VERSION of [5]
++ PATH_TO_JAVA=/usr/java/jdk1.7.0_67-cloudera/bin/java
+++ /usr/java/jdk1.7.0_67-cloudera/bin/java -version
+++ awk -F '"' '/version/ {print $2}'
++ JAVA_VER=1.7.0_67
++ JAVA_VER_SHORT=1.7
++ [[ 1.7 != \1\.\8 ]]
++ echo 'Java version 1.8 is required for Spark 2.2.'
++ exit 1

[#4]

Hi @Huriye

Thank you for posting your query with Cloudera community.

From the documentation I could see, the requirement for spark 2.2.x is JDK 1.8

Please refer the below URL

[1]. https://www.cloudera.com/documentation/spark2/2-2-x/topics/spark2_requirements.html

[#5]

Hello,

Thank you for your answer, I tried to change java version but cloudera service not starting.

I followed th link: https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cdh_cm_upgrading_to_jdk8.html

But I get error which is Failed to start LSB: Cloudera SCM Server and there is no error on /var/log/cloudera-scm-server

Best regards,

Huriye

[#6 · Accepted solution]

Ok, from the log it is obvioues that issue for spark is the old jdk.

When you tried to upgrade java have you defined the java home in "/etc/default/cloudera-scm-server"

e.g.:

export JAVA_HOME="/usr/lib/jvm/java-8-oracle/"

Can you send the relevant "/var/log/cloudera-scm-server/cloudera-scm-server.out" ?