Skip to main content
R_SHETH
R_SHETH
Silver ⭐️⭐️
August 20, 2019
Solved

spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

[root@sxd-cdh-nn parcels]# spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0-cdh6.1.1
/_/

Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201
Branch HEAD
Compiled by user jenkins on 2019-02-08T07:10:49Z
Revision HEAD
Url
Type --help for more information.

 

 

we have on top of this installed spark2 with 2.2 

 

[root@sxd-cdh-nn parcels]# spark2-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.0.cloudera1
/_/

Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201
Branch HEAD
Compiled by user jenkins on 2017-07-13T00:28:58Z
Revision 39f5a2b89d29d5d420d88ce15c8c55e2b45aeb2e
Url git://github.mtv.cloudera.com/CDH/spark.git
Type --help for more information.

 

Could you please let me know how to uninstall spark-submit --version to 2.4?

 

we only want spark2 with 2.2 version 

 

thanks

Romesh

benhadoop
benhadoopAnswer
Gold ⭐️
August 20, 2019

Based on your output (auto mode in alternatives) you can try to add it with a higher priority (the 2.4 spark-submit has priority 10):

/usr/sbin/alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/SPARK2/bin/spark2-submit 100

 

Be aware that Cloudera Manager might (try to) overwrite this on CDH updates.

 

Again, I would recommend using the packaged Spark that comes with CDH and have doubts that using CDS in CDH 6.x is supported (also see [CDS Requirements - CDH Versions] [Migrating Apache Spark Before Upgrading to CDH 6])


Regards
Benjamin

View in thread

5 replies

benhadoop
benhadoop
Gold ⭐️
August 20, 2019

You can override the "spark-submit" association with alternatives, e.g.:

/usr/sbin/alternatives --set spark-submit <path-to-spark2-submit>

# <path-to-spark2-submit> could be like "/opt/cloudera/parcels/SPARK2-2.2.0-cloudera1-cdh5.13.3.p0.611179/bin/spark2-submit"
# This is a path of CDH 5 to the Spark parcel directory. You need to adjust it to your path (the one that spark2-shell is pointing to)
# You can find this path with:
/usr/sbin/alternatives --display spark2-submit

 

However, from CDH 6.x, it is normal to use spark-submit instead of spark2-submit because there is only Spark2 included in CDH. Also, it is normal to use the packaged version inside of the CDH distribution, which seems to be 2.4.0 for your CDH 6.1.1. I am not sure, if using a different version is supported or recommended by Cloudera.

 

How did you install that Spark 2.2 version in your CDH 6 cluster? 

 

Also see https://www.cloudera.com/documentation/enterprise/6/6.1/topics/spark.html

Translate
R_SHETH
R_SHETHAuthor
Silver ⭐️⭐️
August 20, 2019

Thank you ....

 

this has not been setup by that command. 

 

[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --set spark-submit "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit"
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit has not been configured as an alternative for spark-submit
[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --display spark-submit
spark-submit - status is auto.
link currently points to /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit
/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit - priority 10
Current `best' version is /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit.
[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --display spark2-submit
spark2-submit - status is auto.
link currently points to /opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit - priority 10
Current `best' version is /opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit.
[root@sxd-cdh-nn sbin]#

 

we did not pre-configured Spark2 while installing CDH and now we have installed spark 2.2 via parcels method. 

 

thanks

Romesh 

Translate
benhadoop
benhadoopAnswer
Gold ⭐️
August 20, 2019

Based on your output (auto mode in alternatives) you can try to add it with a higher priority (the 2.4 spark-submit has priority 10):

/usr/sbin/alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/SPARK2/bin/spark2-submit 100

 

Be aware that Cloudera Manager might (try to) overwrite this on CDH updates.

 

Again, I would recommend using the packaged Spark that comes with CDH and have doubts that using CDS in CDH 6.x is supported (also see [CDS Requirements - CDH Versions] [Migrating Apache Spark Before Upgrading to CDH 6])


Regards
Benjamin

Translate
Terms & ConditionsAccessibility statement