how to upgrade spark 1.3.1.2.3 to spark 1.41

[#1]

Versions Installed :

HDP-2.3.6.0-3796

Spark 1.3.1.2.3

HDFS 2.7.1.2.3

Ambari 2.1.0

Currently we are integrating SAP HANA with Hadoop and due to versions compatibility we have to move to only Spark 1.4.1.

At this point of time we have Spark 1.3.1.2.3 so we need to upgrade to Spark 1.4.1

[#2 · Accepted solution]

@Zeeshan Ahmed

Stop & Delete Spark Ambari service:

$ curl -u admin:admin
-H "X-Requested-By:ambari" -i PUT -d '{"RequestInfo":{"context":"Stop
Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}'
http://AMBARI-URL:8080//api/v1/clusters/CLUSTERNAME/services/SPARK 

$ curl -u admin:admin
-H "X-Requested-By:ambari" -X DELETE http://AMBARI-URL:8080/api/v1/clusters/CLUSTERNAME/services/SPARK

Stop Spark 1.3.1 history server:

su
- spark -c
"/usr/hdp/current/spark-client/sbin/stop-history-server.sh"

Remove Spark 1.3.1:

yum erase "spark*"

Add the node where you want Spark 1.4.1 History Server and Client to run:

$ su - root wget -nv
$ http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.3.2.0-2950/hdpbn.repo
-O /etc/yum.repos.d/Spark141TP.repo 

$ yum install spark_2_3_2_0_2950-master -y 

$ conf-select create-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 cp
/etc/spark/2.3.0.0-2950/0/* /etc/spark/2.3.2.0-2950/0/ 

$ conf-select set-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 

$ hdp-select set spark-client 2.3.2.0-2950 

$ hdp-select set spark-historyserver 2.3.2.0-2950

Validate the Spark installation. As user spark, run SparkPI example:

 su - spark -c
"cd /usr/hdp/current/spark-client" ./bin/spark-submit
--class org.apache.spark.examples.SparkPi --master yarn-client --num-executors
3 --driver-memory 512m --executor-memory 512m --executor-cores 1
lib/spark-examples*.jar 10

[#3]

Hi @Vedant Jain ,

I performed the above steps and successfully removed spark 1.3.1 and installed spark 1.4.1, but in Ambari I don't see the newly Added Spark service. The old spark in not visible anymore.

Due to project needs I need to see the current spark service in my Ambari. Is there any way I can achieve this ?

Thanks in Advance.

[#4]

@Zeeshan Ahmed

This needs to be done after the step: remove version 1.3.1

I personally have not tried it, but you can download the version of Spark you are looking for from here:

https://github.com/apache/ambari/tree/2ad42074f1633c5c6f56cf979bdaa49440457566/ambari-server/src/mai...

Create a directory called SPARK in /var/lib/ambari-server/resources/stacks/HDP/2.3/services/

and copy the downloaded contents into this directory.

Restart ambari server and you should see this version of Spark as an option in 'Add a service' menu in Ambari. Follow the steps to install the service.