Support Questions
Find answers, ask questions, and share your expertise

How to use Livy API call Synchronously to exe Spark Java jar on AWS EMR

New Contributor

I am trying to add Spark Jar to emr cluster with Livy API but I want to do it in Sync mode.

I went through bunch of articles and came across LivyClient but that requires me to change Spark Job code in order to make it work.

Is there a way to get it done?

I have below Spark java code.

SparkConf conf = new SparkConf().setAppName("testapp");
JavaSparkContext sc = new JavaSparkContext(conf);

JavaRDD<String> lines = sc.textFile(args[0]);
JavaRDD<String> numbers = lines.flatMap(line -> Arrays.asList(line.split("\\s+")).iterator());

JavaRDD<String> validNumbers = numbers.filter(number -> !number.isEmpty());

JavaRDD<Integer> intNumbers = validNumbers.map(number -> Integer.valueOf(number));

System.out.println("Sum is: " + intNumbers.reduce((x, y) -> x + y));

intNumbers.saveAsTextFile(args[1]);
0 REPLIES 0