Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark YARN 1.1.0 Release

Spark YARN 1.1.0 Release

New Contributor

Hello,

 

When will the following artifact be released (changed to a non-snapshot version) and pushed to the repo?

 

https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-yarn_2.10/1.1.0-cd...

 

Thank you.

 

Mik

 

3 REPLIES 3

Re: Spark YARN 1.1.0 Release

New Contributor

Anyone haven an ETA on this?

Re: Spark YARN 1.1.0 Release

Master Collaborator

Looks like this slipped through the cracks. Word from expert Marcelo is:

 

  • there's typically no reason to depend on this module because it does not have a public API, and
  • the YARN modules will no longer be published by Spark, as of Spark 1.2.0, and
  • they were actually disabled a while ago but not for 1.1.x, but
  • looks like we took the change that disables it into our 1.1.x release kind of accidentally

Practically speaking, the best solution for you would be to not use these, as they were not really intended to be published by Spark, but is there a use case for directly depending on the module?

Re: Spark YARN 1.1.0 Release

New Contributor

I tracked down the changes to Spark 1.2 and indeed they made them private. I linked to the issue if anyone wants to monitor that and my request to "publicate"... "publicize"... to make them public again.

 

The reason we are depending on the module is to use the Spark 1.1 public facing org.apache.spark.deploy.yarn.Client. We are attempting to emulate the behavior of the spark-submit for execution in a Java appliction by creating a Client and executing the runApp() method.

 

We are trying to avoid making system call to a script from Java by doing this. Also, not using spark-submit means we don't have to setup a complicated environment e.g. hadoop xml files and environment variables.

 

If anyone has suggestions that would be super. We tried the spark-jobserver but that has some issues with using YARN instead of Mesos.

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here