Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

avatar
Super Collaborator

Currently , we are using Spark 2.0 on HDP 2.5.3 and its working absolutely fine, when i submit the job manually. We want to automate the same using OOZIE and for the same we require spark-assembly jar for spark 2.0, unfortunately can't find the same in /usr/hdp/current/spark2-client/jars/

Unable to find the same in Maven Repository also.

1 ACCEPTED SOLUTION

avatar
Master Mentor

Spark2 action in Oozie is not supported in HDP 2.5 or earlier. Spark2 action will ship as part of Oozie 5.0 which is still unreleased by community. HDP is on Oozie 4.2, there are discussions to bring Spark2 into Oozie sooner.

View solution in original post

5 REPLIES 5

avatar
Super Collaborator

I see that spark-assembly jar is no longer required for spark 2.0 but when we are using spark action , receiving the below error

ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
	at comm.common.Acct$.getAcctDF(Acct.scala:13)
	at comm.billlogic.SparkExecute$.execute(SparkExecute.scala:45)
	at comm.billlogic.BillLogic$.main(BillLogic.scala:26)
	at comm.billlogic.BillLogic.main(BillLogic.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.spark.deploy.yarn.ApplicationMaster$anon$2.run(ApplicationMaster.scala:559)

avatar
Master Mentor

@Krishna Srinivas

The spark-assembly jar can be found in the following location:

# ls -l /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar 
-rw-r--r--. 1 root root 188897932 Nov 30 04:01 /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar

# Which is a symlink
# ls -lnrt /usr/hdp/2.5.3.0-37/spark/lib/spark-hdp-assembly.jar 
lrwxrwxrwx. 1 0 0 88 Jan 25 07:16 /usr/hdp/2.5.3.0-37/spark/lib/spark-hdp-assembly.jar -> /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar

.

However the next error that you are getting as following indicates that you have multiple version of Spark JARs present in your classpath. Can you please remove the other version of spark jars and then try?

java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;

.

avatar
Super Collaborator

@Jay SenSharma HI Jay, Currently we have Spark 1.6 and 2.0 both installed on the cluster , and currently we are using spark 2.0 for our operations by using

export SPARK_MAJOR_VERSION=2

After that, if we use Spark-submit command there was no issue and it is successful, unfortunately Issue is with Oozie only

avatar
Master Mentor

Spark2 action in Oozie is not supported in HDP 2.5 or earlier. Spark2 action will ship as part of Oozie 5.0 which is still unreleased by community. HDP is on Oozie 4.2, there are discussions to bring Spark2 into Oozie sooner.

avatar
New Contributor

@klksrinivas Did find a solution how to run a job (Spark2) with Oozie?