Question on Spark Versioning

[#1]

I am little confused while I am checking the version of spark :

ON the server it says as follows :

/usr/hdp/2.3.0.0-2557/spark

[admin2.prod.iad spark]$ more RELEASE

Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557

Whereas on Ambari it says as follows :

Spark

1.5.2.2.3

Installed

Apache Spark is a fast and general engine for large-scale data processing.

which one is the right one.

Also on Ambari the status says "Installed", what does this mean ?

[#2]

@Kumar Veerappan

This is confusing. Not sure what's causing it. But to find the right spark version, just run your spark-shell. It shows spark version when it starts.

[#3]

@Kumar Veerappan

I see that might be problem. you can use below command to check correct version of spark and relate to your above outputs -

$rpm -qa |grep spark

As i see in HDP 2.4.2 spark version is correctly displayed as shown below -

[#4]

the spark shell says as follows :

Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.3.1 /_/ Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_99)

so I believe this is the right one

[#5]

yes, you are using version 1.3.1.

[#6 · Accepted solution]

@Kumar Veerappan

1.3.1 is the Spark version supported by HDP 2.3.0.

Would it be possible that someone installed a newer version of Spark outside of Ambari then uninstalled and Ambari is caching somehow that version. Did you restart Ambari server and checked again?