Reply
mik
New Contributor
Posts: 2
Registered: ‎11-18-2014

Spark YARN 1.1.0 Release

Hello,

 

When will the following artifact be released (changed to a non-snapshot version) and pushed to the repo?

 

https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-yarn_2.10/1.1.0-cd...

 

Thank you.

 

Mik

 

mik
New Contributor
Posts: 2
Registered: ‎11-18-2014

Re: Spark YARN 1.1.0 Release

Anyone haven an ETA on this?

Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: Spark YARN 1.1.0 Release

Looks like this slipped through the cracks. Word from expert Marcelo is:

 

  • there's typically no reason to depend on this module because it does not have a public API, and
  • the YARN modules will no longer be published by Spark, as of Spark 1.2.0, and
  • they were actually disabled a while ago but not for 1.1.x, but
  • looks like we took the change that disables it into our 1.1.x release kind of accidentally

Practically speaking, the best solution for you would be to not use these, as they were not really intended to be published by Spark, but is there a use case for directly depending on the module?

Highlighted
New Contributor
Posts: 3
Registered: ‎12-09-2014

Re: Spark YARN 1.1.0 Release

I tracked down the changes to Spark 1.2 and indeed they made them private. I linked to the issue if anyone wants to monitor that and my request to "publicate"... "publicize"... to make them public again.

 

The reason we are depending on the module is to use the Spark 1.1 public facing org.apache.spark.deploy.yarn.Client. We are attempting to emulate the behavior of the spark-submit for execution in a Java appliction by creating a Client and executing the runApp() method.

 

We are trying to avoid making system call to a script from Java by doing this. Also, not using spark-submit means we don't have to setup a complicated environment e.g. hadoop xml files and environment variables.

 

If anyone has suggestions that would be super. We tried the spark-jobserver but that has some issues with using YARN instead of Mesos.