11-18-2014 03:51 PM
When will the following artifact be released (changed to a non-snapshot version) and pushed to the repo?
12-09-2014 10:40 AM
Looks like this slipped through the cracks. Word from expert Marcelo is:
Practically speaking, the best solution for you would be to not use these, as they were not really intended to be published by Spark, but is there a use case for directly depending on the module?
12-09-2014 01:24 PM
I tracked down the changes to Spark 1.2 and indeed they made them private. I linked to the issue if anyone wants to monitor that and my request to "publicate"... "publicize"... to make them public again.
The reason we are depending on the module is to use the Spark 1.1 public facing org.apache.spark.deploy.yarn.Client. We are attempting to emulate the behavior of the spark-submit for execution in a Java appliction by creating a Client and executing the runApp() method.
We are trying to avoid making system call to a script from Java by doing this. Also, not using spark-submit means we don't have to setup a complicated environment e.g. hadoop xml files and environment variables.
If anyone has suggestions that would be super. We tried the spark-jobserver but that has some issues with using YARN instead of Mesos.