Support Questions
Find answers, ask questions, and share your expertise

Needed spark-assembly-1.5.2.-hadoop2.7.jar for Spark -Oozie workflow on HDP 2.3.2

I am trying to schedule spark 1.5.2 job on oozie 4.2.0 (HDP 2.3.x). Spark 1.5.2 has been installed externally i am not using default spark version provided by hortonworks. I am referring below post to set this up.

https://community.hortonworks.com/questions/7014/oozie-sparkaction-throwing-javalangnosuchmethoderr....

I am struggling to find below jars.

-spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

-spark-examples-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

If you can help me with some pointers to find/download above jars it will be a great help to get started.

3 REPLIES 3

I assume you have access to a HDP 2.3.4 stack. For me the files are at

/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar
/usr/hdp/2.3.4.0-3485/spark/lib/spark-examples-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar

@Bernhard Walter As mentioned above Spark 1.5.2 has been installed externally i am not using default spark version 1.4.1 provided by hortonworks. The location you specified above have jar for spark version 1.4.1. I need assembly jar for spark 1.5.2 and hadoop 2.7.x

Then you most probably need to use the assemblies of this version instead of the HDP 2.3.4 ones.

Assuming you downloaded Spark 1.5.2 from spark webpage then you might want to try the assembly and examples that comes with this version instead

spark-1.5.2-bin-hadoop2.6/lib/spark-assembly-1.5.2-hadoop2.6.0.jar
spark-1.5.2-bin-hadoop2.6/lib/spark-examples-1.5.2-hadoop2.6.0.jar