Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

avatar
New Contributor

[root@sxd-cdh-nn parcels]# spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0-cdh6.1.1
/_/

Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201
Branch HEAD
Compiled by user jenkins on 2019-02-08T07:10:49Z
Revision HEAD
Url
Type --help for more information.

 

 

we have on top of this installed spark2 with 2.2 

 

[root@sxd-cdh-nn parcels]# spark2-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.0.cloudera1
/_/

Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201
Branch HEAD
Compiled by user jenkins on 2017-07-13T00:28:58Z
Revision 39f5a2b89d29d5d420d88ce15c8c55e2b45aeb2e
Url git://github.mtv.cloudera.com/CDH/spark.git
Type --help for more information.

 

Could you please let me know how to uninstall spark-submit --version to 2.4?

 

we only want spark2 with 2.2 version 

 

thanks

Romesh

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Based on your output (auto mode in alternatives) you can try to add it with a higher priority (the 2.4 spark-submit has priority 10):

/usr/sbin/alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/SPARK2/bin/spark2-submit 100

 

Be aware that Cloudera Manager might (try to) overwrite this on CDH updates.

 

Again, I would recommend using the packaged Spark that comes with CDH and have doubts that using CDS in CDH 6.x is supported (also see [CDS Requirements - CDH Versions] [Migrating Apache Spark Before Upgrading to CDH 6])


Regards
Benjamin

View solution in original post

5 REPLIES 5

avatar
Expert Contributor

You can override the "spark-submit" association with alternatives, e.g.:

/usr/sbin/alternatives --set spark-submit <path-to-spark2-submit>

# <path-to-spark2-submit> could be like "/opt/cloudera/parcels/SPARK2-2.2.0-cloudera1-cdh5.13.3.p0.611179/bin/spark2-submit"
# This is a path of CDH 5 to the Spark parcel directory. You need to adjust it to your path (the one that spark2-shell is pointing to)
# You can find this path with:
/usr/sbin/alternatives --display spark2-submit

 

However, from CDH 6.x, it is normal to use spark-submit instead of spark2-submit because there is only Spark2 included in CDH. Also, it is normal to use the packaged version inside of the CDH distribution, which seems to be 2.4.0 for your CDH 6.1.1. I am not sure, if using a different version is supported or recommended by Cloudera.

 

How did you install that Spark 2.2 version in your CDH 6 cluster? 

 

Also see https://www.cloudera.com/documentation/enterprise/6/6.1/topics/spark.html

avatar
New Contributor

Thank you ....

 

this has not been setup by that command. 

 

[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --set spark-submit "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit"
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit has not been configured as an alternative for spark-submit
[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --display spark-submit
spark-submit - status is auto.
link currently points to /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit
/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit - priority 10
Current `best' version is /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit.
[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --display spark2-submit
spark2-submit - status is auto.
link currently points to /opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit - priority 10
Current `best' version is /opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit.
[root@sxd-cdh-nn sbin]#

 

we did not pre-configured Spark2 while installing CDH and now we have installed spark 2.2 via parcels method. 

 

thanks

Romesh 

avatar
Expert Contributor

Based on your output (auto mode in alternatives) you can try to add it with a higher priority (the 2.4 spark-submit has priority 10):

/usr/sbin/alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/SPARK2/bin/spark2-submit 100

 

Be aware that Cloudera Manager might (try to) overwrite this on CDH updates.

 

Again, I would recommend using the packaged Spark that comes with CDH and have doubts that using CDS in CDH 6.x is supported (also see [CDS Requirements - CDH Versions] [Migrating Apache Spark Before Upgrading to CDH 6])


Regards
Benjamin

avatar
New Contributor

Thank you Benjamin.. This has fixed the issue. 

 

I would take a note on upgrade point. 

 

thanks

Romesh

avatar
Expert Contributor

Hi @R_SHETH, great to hear!

Please mark the reply as the accepted answer if it solved your problem 🙂