Interested to hear what others are doing about deploying Spark applications to their clusters.
Currently I use Oozie to manage MapReduce / Hive workflows. It's not perfect (far from it), but at least the Hue GUI offers a nice view of the workflow and clearly indicates when a stage has failed.
Am interested to hear what people are doing in Spark-land. Currently I've got a Spark application running nightly. I'm using Oozie to run a shell script that runs the Spark script with: spark-shell < myscript.scala
That's about as nasty as it gets. I can think of a couple of alternatives:
Is there an alternative? What do others do?