Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

Solved Go to solution
Highlighted

unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

Expert Contributor

Currently , we are using Spark 2.0 on HDP 2.5.3 and its working absolutely fine, when i submit the job manually. We want to automate the same using OOZIE and for the same we require spark-assembly jar for spark 2.0, unfortunately can't find the same in /usr/hdp/current/spark2-client/jars/

Unable to find the same in Maven Repository also.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

Mentor

Spark2 action in Oozie is not supported in HDP 2.5 or earlier. Spark2 action will ship as part of Oozie 5.0 which is still unreleased by community. HDP is on Oozie 4.2, there are discussions to bring Spark2 into Oozie sooner.

5 REPLIES 5

Re: unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

Expert Contributor

I see that spark-assembly jar is no longer required for spark 2.0 but when we are using spark action , receiving the below error

ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
	at comm.common.Acct$.getAcctDF(Acct.scala:13)
	at comm.billlogic.SparkExecute$.execute(SparkExecute.scala:45)
	at comm.billlogic.BillLogic$.main(BillLogic.scala:26)
	at comm.billlogic.BillLogic.main(BillLogic.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.spark.deploy.yarn.ApplicationMaster$anon$2.run(ApplicationMaster.scala:559)

Re: unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

Super Mentor

@Krishna Srinivas

The spark-assembly jar can be found in the following location:

# ls -l /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar 
-rw-r--r--. 1 root root 188897932 Nov 30 04:01 /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar

# Which is a symlink
# ls -lnrt /usr/hdp/2.5.3.0-37/spark/lib/spark-hdp-assembly.jar 
lrwxrwxrwx. 1 0 0 88 Jan 25 07:16 /usr/hdp/2.5.3.0-37/spark/lib/spark-hdp-assembly.jar -> /usr/hdp/2.5.3.0-37/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar

.

However the next error that you are getting as following indicates that you have multiple version of Spark JARs present in your classpath. Can you please remove the other version of spark jars and then try?

java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;

.

Re: unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

Expert Contributor

@Jay SenSharma HI Jay, Currently we have Spark 1.6 and 2.0 both installed on the cluster , and currently we are using spark 2.0 for our operations by using

export SPARK_MAJOR_VERSION=2

After that, if we use Spark-submit command there was no issue and it is successful, unfortunately Issue is with Oozie only

Re: unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

Mentor

Spark2 action in Oozie is not supported in HDP 2.5 or earlier. Spark2 action will ship as part of Oozie 5.0 which is still unreleased by community. HDP is on Oozie 4.2, there are discussions to bring Spark2 into Oozie sooner.

Re: unable to find spark-assembly jar for SPARK 2.0 on HDP 2.5.3

New Contributor

@klksrinivas Did find a solution how to run a job (Spark2) with Oozie?

Don't have an account?
Coming from Hortonworks? Activate your account here