Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to upgrade spark 1.3.1.2.3 to spark 1.41

Solved Go to solution

how to upgrade spark 1.3.1.2.3 to spark 1.41

Contributor

Versions Installed :

HDP-2.3.6.0-3796

Spark 1.3.1.2.3

HDFS 2.7.1.2.3

Ambari 2.1.0

Currently we are integrating SAP HANA with Hadoop and due to versions compatibility we have to move to only Spark 1.4.1.

At this point of time we have Spark 1.3.1.2.3 so we need to upgrade to Spark 1.4.1

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: how to upgrade spark 1.3.1.2.3 to spark 1.41

Guru

@Zeeshan Ahmed

Stop & Delete Spark Ambari service:

$ curl -u admin:admin
-H "X-Requested-By:ambari" -i PUT -d '{"RequestInfo":{"context":"Stop
Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}'
http://AMBARI-URL:8080//api/v1/clusters/CLUSTERNAME/services/SPARK 

$ curl -u admin:admin
-H "X-Requested-By:ambari" -X DELETE http://AMBARI-URL:8080/api/v1/clusters/CLUSTERNAME/services/SPARK

Stop Spark 1.3.1 history server:

su
- spark -c
"/usr/hdp/current/spark-client/sbin/stop-history-server.sh"

Remove Spark 1.3.1:

yum erase "spark*"

Add the node where you want Spark 1.4.1 History Server and Client to run:

$ su - root wget -nv
$ http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.3.2.0-2950/hdpbn.repo
-O /etc/yum.repos.d/Spark141TP.repo 

$ yum install spark_2_3_2_0_2950-master -y 

$ conf-select create-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 cp
/etc/spark/2.3.0.0-2950/0/* /etc/spark/2.3.2.0-2950/0/ 

$ conf-select set-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 

$ hdp-select set spark-client 2.3.2.0-2950 

$ hdp-select set spark-historyserver 2.3.2.0-2950

Validate the Spark installation. As user spark, run SparkPI example:

 su - spark -c
"cd /usr/hdp/current/spark-client" ./bin/spark-submit
--class org.apache.spark.examples.SparkPi --master yarn-client --num-executors
3 --driver-memory 512m --executor-memory 512m --executor-cores 1
lib/spark-examples*.jar 10
3 REPLIES 3
Highlighted

Re: how to upgrade spark 1.3.1.2.3 to spark 1.41

Guru

@Zeeshan Ahmed

Stop & Delete Spark Ambari service:

$ curl -u admin:admin
-H "X-Requested-By:ambari" -i PUT -d '{"RequestInfo":{"context":"Stop
Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}'
http://AMBARI-URL:8080//api/v1/clusters/CLUSTERNAME/services/SPARK 

$ curl -u admin:admin
-H "X-Requested-By:ambari" -X DELETE http://AMBARI-URL:8080/api/v1/clusters/CLUSTERNAME/services/SPARK

Stop Spark 1.3.1 history server:

su
- spark -c
"/usr/hdp/current/spark-client/sbin/stop-history-server.sh"

Remove Spark 1.3.1:

yum erase "spark*"

Add the node where you want Spark 1.4.1 History Server and Client to run:

$ su - root wget -nv
$ http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.3.2.0-2950/hdpbn.repo
-O /etc/yum.repos.d/Spark141TP.repo 

$ yum install spark_2_3_2_0_2950-master -y 

$ conf-select create-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 cp
/etc/spark/2.3.0.0-2950/0/* /etc/spark/2.3.2.0-2950/0/ 

$ conf-select set-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 

$ hdp-select set spark-client 2.3.2.0-2950 

$ hdp-select set spark-historyserver 2.3.2.0-2950

Validate the Spark installation. As user spark, run SparkPI example:

 su - spark -c
"cd /usr/hdp/current/spark-client" ./bin/spark-submit
--class org.apache.spark.examples.SparkPi --master yarn-client --num-executors
3 --driver-memory 512m --executor-memory 512m --executor-cores 1
lib/spark-examples*.jar 10

Re: how to upgrade spark 1.3.1.2.3 to spark 1.41

Contributor

Hi @Vedant Jain ,

I performed the above steps and successfully removed spark 1.3.1 and installed spark 1.4.1, but in Ambari I don't see the newly Added Spark service. The old spark in not visible anymore.

Due to project needs I need to see the current spark service in my Ambari. Is there any way I can achieve this ?

Thanks in Advance.

Re: how to upgrade spark 1.3.1.2.3 to spark 1.41

Guru

@Zeeshan Ahmed

This needs to be done after the step: remove version 1.3.1

I personally have not tried it, but you can download the version of Spark you are looking for from here:

https://github.com/apache/ambari/tree/2ad42074f1633c5c6f56cf979bdaa49440457566/ambari-server/src/mai...

Create a directory called SPARK in /var/lib/ambari-server/resources/stacks/HDP/2.3/services/

and copy the downloaded contents into this directory.

Restart ambari server and you should see this version of Spark as an option in 'Add a service' menu in Ambari. Follow the steps to install the service.

Don't have an account?
Coming from Hortonworks? Activate your account here