question Re: Question on Spark Versioning in Archives of Support Questions (Read Only)

Question on Spark Versioning

kums — Wed, 03 Aug 2016 23:20:54 GMT

I am little confused while I am checking the version of spark :

ON the server it says as follows :

/usr/hdp/2.3.0.0-2557/spark

[admin2.prod.iad spark]$ more RELEASE

Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557

Whereas on Ambari it says as follows :

Spark

1.5.2.2.3

Installed

Apache Spark is a fast and general engine for large-scale data processing.

which one is the right one.

Also on Ambari the status says "Installed", what does this mean ?

Re: Question on Spark Versioning

mqureshi — Wed, 03 Aug 2016 23:38:32 GMT

@Kumar Veerappan

This is confusing. Not sure what's causing it. But to find the right spark version, just run your spark-shell. It shows spark version when it starts.

Re: Question on Spark Versioning

sshimpi — Sun, 18 Aug 2019 11:32:37 GMT

@Kumar Veerappan

I see that might be problem. you can use below command to check correct version of spark and relate to your above outputs -

$rpm -qa |grep spark

As i see in HDP 2.4.2 spark version is correctly displayed as shown below -

Re: Question on Spark Versioning

kums — Wed, 03 Aug 2016 23:55:06 GMT

the spark shell says as follows :

Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.3.1 /_/ Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_99)

so I believe this is the right one

Re: Question on Spark Versioning

mqureshi — Thu, 04 Aug 2016 00:00:23 GMT

yes, you are using version 1.3.1.

Re: Question on Spark Versioning

cstanca — Wed, 21 Sep 2016 21:32:59 GMT

@Kumar Veerappan

1.3.1 is the Spark version supported by HDP 2.3.0.

Would it be possible that someone installed a newer version of Spark outside of Ambari then uninstalled and Ambari is caching somehow that version. Did you restart Ambari server and checked again?