Hi, Im currently using Oozie to run a Spark Program developed in Scala. Now, I want to execute a simple unix command using Spark Action Node such as :
echo "name=\"Yvan\""
Once executed by the spark program, I'll need to capture the echoed value in the spark program to be used in the Oozie Workflow.
Essentially, using the shell action command with the following value:
echo "name=\"Yvan\""
can be easily captured using this argument across different action nodes in the workflow:
${wf:actionData('spark-XXX')['name']}
Now, If i just execute a shell command from the spark program, It doesn't seem to be able to capture the values properly. using this spark program code:
import java.lang.ProcessBuilder
import java.io._
import Util.SparkHelper
import sys.process._
object Printer {
def main(args: Array[String]) {
val helper = new Util.SparkHelper()
val sparkContext = helper.getSparkContext("local", "LoadHiveTable")
val varname = args(0)
val value = args(1)
var command = new ProcessBuilder("echo", varname + "=" + "\"" + value + "\"");
command.start()
}
}
Whenever i run this spark program in the oozie workflow, It doesn't seem to properly capture the echoed command from the spark program. Is there a way to properly "capture" the executed echoed value from the spark program to be used as an argument/variable across different action nodes in the workflow?
Any help would be really appreciated. Thank you.
