Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Needed spark-assembly-1.5.2.-hadoop2.7.jar for Spark -Oozie workflow on HDP 2.3.2

Highlighted

Needed spark-assembly-1.5.2.-hadoop2.7.jar for Spark -Oozie workflow on HDP 2.3.2

I am trying to schedule spark 1.5.2 job on oozie 4.2.0 (HDP 2.3.x). Spark 1.5.2 has been installed externally i am not using default spark version provided by hortonworks. I am referring below post to set this up.

https://community.hortonworks.com/questions/7014/oozie-sparkaction-throwing-javalangnosuchmethoderr....

I am struggling to find below jars.

-spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

-spark-examples-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

If you can help me with some pointers to find/download above jars it will be a great help to get started.

3 REPLIES 3
Highlighted

Re: Needed spark-assembly-1.5.2.-hadoop2.7.jar for Spark -Oozie workflow on HDP 2.3.2

I assume you have access to a HDP 2.3.4 stack. For me the files are at

/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar
/usr/hdp/2.3.4.0-3485/spark/lib/spark-examples-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar
Highlighted

Re: Needed spark-assembly-1.5.2.-hadoop2.7.jar for Spark -Oozie workflow on HDP 2.3.2

@Bernhard Walter As mentioned above Spark 1.5.2 has been installed externally i am not using default spark version 1.4.1 provided by hortonworks. The location you specified above have jar for spark version 1.4.1. I need assembly jar for spark 1.5.2 and hadoop 2.7.x

Re: Needed spark-assembly-1.5.2.-hadoop2.7.jar for Spark -Oozie workflow on HDP 2.3.2

Then you most probably need to use the assemblies of this version instead of the HDP 2.3.4 ones.

Assuming you downloaded Spark 1.5.2 from spark webpage then you might want to try the assembly and examples that comes with this version instead

spark-1.5.2-bin-hadoop2.6/lib/spark-assembly-1.5.2-hadoop2.6.0.jar
spark-1.5.2-bin-hadoop2.6/lib/spark-examples-1.5.2-hadoop2.6.0.jar
Don't have an account?
Coming from Hortonworks? Activate your account here