Created 11-15-2016 01:47 PM
Versions Installed :
HDP-2.3.6.0-3796
Spark 1.3.1.2.3
HDFS 2.7.1.2.3
Ambari 2.1.0Currently we are integrating SAP HANA with Hadoop and due to versions compatibility we have to move to only Spark 1.4.1.
At this point of time we have Spark 1.3.1.2.3 so we need to upgrade to Spark 1.4.1
Created 11-15-2016 03:28 PM
Stop & Delete Spark Ambari service:
$ curl -u admin:admin -H "X-Requested-By:ambari" -i PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://AMBARI-URL:8080//api/v1/clusters/CLUSTERNAME/services/SPARK $ curl -u admin:admin -H "X-Requested-By:ambari" -X DELETE http://AMBARI-URL:8080/api/v1/clusters/CLUSTERNAME/services/SPARK
Stop Spark 1.3.1 history server:
su - spark -c "/usr/hdp/current/spark-client/sbin/stop-history-server.sh"
Remove Spark 1.3.1:
yum erase "spark*"
Add the node where you want Spark 1.4.1 History Server and Client to run:
$ su - root wget -nv $ http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.3.2.0-2950/hdpbn.repo -O /etc/yum.repos.d/Spark141TP.repo $ yum install spark_2_3_2_0_2950-master -y $ conf-select create-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 cp /etc/spark/2.3.0.0-2950/0/* /etc/spark/2.3.2.0-2950/0/ $ conf-select set-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 $ hdp-select set spark-client 2.3.2.0-2950 $ hdp-select set spark-historyserver 2.3.2.0-2950
Validate the Spark installation. As user spark, run SparkPI example:
su - spark -c "cd /usr/hdp/current/spark-client" ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10
Created 11-15-2016 03:28 PM
Stop & Delete Spark Ambari service:
$ curl -u admin:admin -H "X-Requested-By:ambari" -i PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://AMBARI-URL:8080//api/v1/clusters/CLUSTERNAME/services/SPARK $ curl -u admin:admin -H "X-Requested-By:ambari" -X DELETE http://AMBARI-URL:8080/api/v1/clusters/CLUSTERNAME/services/SPARK
Stop Spark 1.3.1 history server:
su - spark -c "/usr/hdp/current/spark-client/sbin/stop-history-server.sh"
Remove Spark 1.3.1:
yum erase "spark*"
Add the node where you want Spark 1.4.1 History Server and Client to run:
$ su - root wget -nv $ http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.3.2.0-2950/hdpbn.repo -O /etc/yum.repos.d/Spark141TP.repo $ yum install spark_2_3_2_0_2950-master -y $ conf-select create-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 cp /etc/spark/2.3.0.0-2950/0/* /etc/spark/2.3.2.0-2950/0/ $ conf-select set-conf-dir --package spark --stack-version 2.3.2.0-2950 --conf-version 0 $ hdp-select set spark-client 2.3.2.0-2950 $ hdp-select set spark-historyserver 2.3.2.0-2950
Validate the Spark installation. As user spark, run SparkPI example:
su - spark -c "cd /usr/hdp/current/spark-client" ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10
Created 11-15-2016 04:26 PM
Hi @Vedant Jain ,
I performed the above steps and successfully removed spark 1.3.1 and installed spark 1.4.1, but in Ambari I don't see the newly Added Spark service. The old spark in not visible anymore.
Due to project needs I need to see the current spark service in my Ambari. Is there any way I can achieve this ?
Thanks in Advance.
Created 11-15-2016 04:44 PM
This needs to be done after the step: remove version 1.3.1
I personally have not tried it, but you can download the version of Spark you are looking for from here:
Create a directory called SPARK in /var/lib/ambari-server/resources/stacks/HDP/2.3/services/
and copy the downloaded contents into this directory.
Restart ambari server and you should see this version of Spark as an option in 'Add a service' menu in Ambari. Follow the steps to install the service.