Created 08-05-2016 05:54 AM
I am trying to schedule spark 1.5.2 job on oozie 4.2.0 (HDP 2.3.x). Spark 1.5.2 has been installed externally i am not using default spark version provided by hortonworks. I am referring below post to set this up.
I am struggling to find below jars.
-spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar
-spark-examples-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar
If you can help me with some pointers to find/download above jars it will be a great help to get started.
Created 08-05-2016 08:35 AM
I assume you have access to a HDP 2.3.4 stack. For me the files are at
/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar /usr/hdp/2.3.4.0-3485/spark/lib/spark-examples-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar
Created 08-05-2016 08:50 AM
@Bernhard Walter As mentioned above Spark 1.5.2 has been installed externally i am not using default spark version 1.4.1 provided by hortonworks. The location you specified above have jar for spark version 1.4.1. I need assembly jar for spark 1.5.2 and hadoop 2.7.x
Created 08-05-2016 09:28 AM
Then you most probably need to use the assemblies of this version instead of the HDP 2.3.4 ones.
Assuming you downloaded Spark 1.5.2 from spark webpage then you might want to try the assembly and examples that comes with this version instead
spark-1.5.2-bin-hadoop2.6/lib/spark-assembly-1.5.2-hadoop2.6.0.jar spark-1.5.2-bin-hadoop2.6/lib/spark-examples-1.5.2-hadoop2.6.0.jar