Support Questions

Find answers, ask questions, and share your expertise

Question on Spark Versioning

avatar
Expert Contributor

I am little confused while I am checking the version of spark :

ON the server it says as follows :

/usr/hdp/2.3.0.0-2557/spark

[admin2.prod.iad spark]$ more RELEASE

Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557

Whereas on Ambari it says as follows :

Spark1.5.2.2.3InstalledApache Spark is a fast and general engine for large-scale data processing.

which one is the right one.

Also on Ambari the status says "Installed", what does this mean ?

1 ACCEPTED SOLUTION

avatar
Super Guru

@Kumar Veerappan

1.3.1 is the Spark version supported by HDP 2.3.0.

Would it be possible that someone installed a newer version of Spark outside of Ambari then uninstalled and Ambari is caching somehow that version. Did you restart Ambari server and checked again?

View solution in original post

5 REPLIES 5

avatar
Super Guru

@Kumar Veerappan

This is confusing. Not sure what's causing it. But to find the right spark version, just run your spark-shell. It shows spark version when it starts.

avatar
Super Guru
@Kumar Veerappan

I see that might be problem. you can use below command to check correct version of spark and relate to your above outputs -

$rpm -qa |grep spark

As i see in HDP 2.4.2 spark version is correctly displayed as shown below -

6298-screen-shot-2016-08-03-at-100704-pm.png

6299-screen-shot-2016-08-03-at-100740-pm.png

avatar
Expert Contributor

the spark shell says as follows :

Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.3.1 /_/ Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_99)

so I believe this is the right one

avatar
Super Guru

yes, you are using version 1.3.1.

avatar
Super Guru

@Kumar Veerappan

1.3.1 is the Spark version supported by HDP 2.3.0.

Would it be possible that someone installed a newer version of Spark outside of Ambari then uninstalled and Ambari is caching somehow that version. Did you restart Ambari server and checked again?