Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to use Livy API call Synchronously to exe Spark Java jar on AWS EMR

Highlighted

How to use Livy API call Synchronously to exe Spark Java jar on AWS EMR

New Contributor

I am trying to add Spark Jar to emr cluster with Livy API but I want to do it in Sync mode.

I went through bunch of articles and came across LivyClient but that requires me to change Spark Job code in order to make it work.

Is there a way to get it done?

I have below Spark java code.

SparkConf conf = new SparkConf().setAppName("testapp");
JavaSparkContext sc = new JavaSparkContext(conf);

JavaRDD<String> lines = sc.textFile(args[0]);
JavaRDD<String> numbers = lines.flatMap(line -> Arrays.asList(line.split("\\s+")).iterator());

JavaRDD<String> validNumbers = numbers.filter(number -> !number.isEmpty());

JavaRDD<Integer> intNumbers = validNumbers.map(number -> Integer.valueOf(number));

System.out.println("Sum is: " + intNumbers.reduce((x, y) -> x + y));

intNumbers.saveAsTextFile(args[1]);
Don't have an account?
Coming from Hortonworks? Activate your account here