question Re: spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2 in Support Questions

spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

R_SHETH — Tue, 20 Aug 2019 10:16:44 GMT

[root@sxd-cdh-nn parcels]# spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0-cdh6.1.1
/_/

Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201
Branch HEAD
Compiled by user jenkins on 2019-02-08T07:10:49Z
Revision HEAD
Url
Type --help for more information.

we have on top of this installed spark2 with 2.2

[root@sxd-cdh-nn parcels]# spark2-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.0.cloudera1
/_/

Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201
Branch HEAD
Compiled by user jenkins on 2017-07-13T00:28:58Z
Revision 39f5a2b89d29d5d420d88ce15c8c55e2b45aeb2e
Url git://github.mtv.cloudera.com/CDH/spark.git
Type --help for more information.

Could you please let me know how to uninstall spark-submit --version to 2.4?

we only want spark2 with 2.2 version

thanks

Romesh

Re: spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

benhadoop — Tue, 20 Aug 2019 08:56:29 GMT

You can override the "spark-submit" association with alternatives, e.g.:

/usr/sbin/alternatives --set spark-submit <path-to-spark2-submit> # <path-to-spark2-submit> could be like "/opt/cloudera/parcels/SPARK2-2.2.0-cloudera1-cdh5.13.3.p0.611179/bin/spark2-submit" # This is a path of CDH 5 to the Spark parcel directory. You need to adjust it to your path (the one that spark2-shell is pointing to) # You can find this path with: /usr/sbin/alternatives --display spark2-submit

However, from CDH 6.x, it is normal to use spark-submit instead of spark2-submit because there is only Spark2 included in CDH. Also, it is normal to use the packaged version inside of the CDH distribution, which seems to be 2.4.0 for your CDH 6.1.1. I am not sure, if using a different version is supported or recommended by Cloudera.

How did you install that Spark 2.2 version in your CDH 6 cluster?

Also see https://www.cloudera.com/documentation/enterprise/6/6.1/topics/spark.html

Re: spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

R_SHETH — Tue, 20 Aug 2019 09:19:27 GMT

Thank you ....

this has not been setup by that command.

[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --set spark-submit "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit"
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit has not been configured as an alternative for spark-submit
[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --display spark-submit
spark-submit - status is auto.
link currently points to /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit
/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit - priority 10
Current `best' version is /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/bin/spark-submit.
[root@sxd-cdh-nn sbin]# /usr/sbin/alternatives --display spark2-submit
spark2-submit - status is auto.
link currently points to /opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit - priority 10
Current `best' version is /opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/spark2-submit.
[root@sxd-cdh-nn sbin]#

we did not pre-configured Spark2 while installing CDH and now we have installed spark 2.2 via parcels method.

thanks

Romesh

Re: spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

benhadoop — Tue, 20 Aug 2019 11:30:35 GMT

Based on your output (auto mode in alternatives) you can try to add it with a higher priority (the 2.4 spark-submit has priority 10):

/usr/sbin/alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/SPARK2/bin/spark2-submit 100

Be aware that Cloudera Manager might (try to) overwrite this on CDH updates.

Again, I would recommend using the packaged Spark that comes with CDH and have doubts that using CDS in CDH 6.x is supported (also see [CDS Requirements - CDH Versions] [Migrating Apache Spark Before Upgrading to CDH 6])

Regards
Benjamin

Re: spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

R_SHETH — Tue, 20 Aug 2019 11:52:54 GMT

Thank you Benjamin.. This has fixed the issue.

I would take a note on upgrade point.

thanks

Romesh

Re: spark-submit still pointing to Spark-version although installed Spark2-submit on 2.2

benhadoop — Tue, 20 Aug 2019 12:19:30 GMT

Hi @R_SHETH, great to hear!

Please mark the reply as the accepted answer if it solved your problem 🙂