Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Capture Output from Spark Program that is run using Oozie

Capture Output from Spark Program that is run using Oozie

New Contributor

Hi, Im currently using Oozie to run a Spark Program developed in Scala. Now, I want to execute a simple unix command using Spark Action Node such as :

 

echo "name=\"Yvan\""

 

Once executed by the spark program, I'll need to capture the echoed value in the spark program to be used in the Oozie Workflow.

 

Essentially, using the shell action command with the following value:

 

echo "name=\"Yvan\""

 

can be easily captured using this argument across different action nodes in the workflow:

 

${wf:actionData('spark-XXX')['name']}

 

Now, If i just execute a shell command from the spark program, It doesn't seem to be able to capture the values properly. using this spark program code:

 

import java.lang.ProcessBuilder
import java.io._
import Util.SparkHelper
import sys.process._
object Printer {
def main(args: Array[String]) {
val helper = new Util.SparkHelper()
val sparkContext = helper.getSparkContext("local", "LoadHiveTable")
val varname = args(0)
val value = args(1)
var command = new ProcessBuilder("echo", varname + "=" + "\"" + value + "\"");

command.start()

}
}

 

Whenever i run this spark program in the oozie workflow, It doesn't seem to properly capture the echoed command from the spark program. Is there a way to properly "capture" the executed echoed value from the spark program to be used as an argument/variable across different action nodes in the workflow?

Any help would be really appreciated. Thank you.

 

1 REPLY 1
Highlighted

Re: Capture Output from Spark Program that is run using Oozie

New Contributor
Don't have an account?
Coming from Hortonworks? Activate your account here