unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

[#1]

Currently , we are using Spark 2.0 on HDP 2.5.3 and its working absolutely fine, when i submit the job manually. We want to automate the same using OOZIE and for the same we require spark-assembly jar for spark 2.0, unfortunately can't find the same in /usr/hdp/current/spark2-client/jars/

Unable to find the same in Maven Repository also.

[#2]

I see that spark-assembly jar is no longer required for spark 2.0 but when we are using spark action , receiving the below error

ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
	at comm.common.Acct$.getAcctDF(Acct.scala:13)
	at comm.billlogic.SparkExecute$.execute(SparkExecute.scala:45)
	at comm.billlogic.BillLogic$.main(BillLogic.scala:26)
	at comm.billlogic.BillLogic.main(BillLogic.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.spark.deploy.yarn.ApplicationMaster$anon$2.run(ApplicationMaster.scala:559)

[#3]

@Krishna Srinivas

The spark-assembly jar can be found in the following location:

# ls -l /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar 
-rw-r--r--. 1 root root 188897932 Nov 30 04:01 /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar

# Which is a symlink
# ls -lnrt /usr/hdp/2.5.3.0-37/spark/lib/spark-hdp-assembly.jar 
lrwxrwxrwx. 1 0 0 88 Jan 25 07:16 /usr/hdp/2.5.3.0-37/spark/lib/spark-hdp-assembly.jar -> /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar

.

However the next error that you are getting as following indicates that you have multiple version of Spark JARs present in your classpath. Can you please remove the other version of spark jars and then try?

java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;

.

[#4]

@Jay SenSharma HI Jay, Currently we have Spark 1.6 and 2.0 both installed on the cluster , and currently we are using spark 2.0 for our operations by using

export SPARK_MAJOR_VERSION=2

After that, if we use Spark-submit command there was no issue and it is successful, unfortunately Issue is with Oozie only

[#5 · Accepted solution]

Spark2 action in Oozie is not supported in HDP 2.5 or earlier. Spark2 action will ship as part of Oozie 5.0 which is still unreleased by community. HDP is on Oozie 4.2, there are discussions to bring Spark2 into Oozie sooner.

[#6]

@klksrinivas Did find a solution how to run a job (Spark2) with Oozie?