Support Questions
Find answers, ask questions, and share your expertise

Spark launching in local mode but not on yarn clusrer

Spark launching in local mode but not on yarn clusrer

Hi,

I have an issue on a Yarn cluster. I am able to run my application in local mode on the entry/main node in the cluster but when I am launching it on cluster (using client or cluster mode) it just does not start on the cluster. The error is simply that yarn container is not launching with an exit code of -1. What could be the issue?

Tried many things and surprisingly the configuration on this cluster is same as another independent cluster where the Spark application is running in cluster mode.

4 REPLIES 4

Re: Spark launching in local mode but not on yarn clusrer

Contributor

Can you share an example of how you are trying to run the job? Is it a java or pyspark?

From the documentation: HDP 2.4 Spark Pi Program

For Java

spark-submit --master yarn-client --class <java.class.getting.called> something.jar

For PytSpark

spark-submit --master yarn-client something.py

Re: Spark launching in local mode but not on yarn clusrer

spark-submit --class "com.MyClass" --master yarn --deploy-mode client --driver-memory 4g --executor-memory 4g --num-executors 4 --executor-cores 10 logminer-1.0.jar -arg1 value1

Re: Spark launching in local mode but not on yarn clusrer

Contributor

To better help you with this, are you able to run the example program that is listed in the document link I showed up above? I suspect that either you are not able to submit a job because the node you are on does not have a spark client installed or the environment variables are not propperly getting sourced.

One thing that might help is if you first execute the environment shell script: (on RHEL/CentOS)

# sh /usr/hdp/current-version/spark-client/conf/spark-env.sh

Then try running the example like in the docs and see if it succeeds. If it fails, we can look more closely at what the issue is.

Re: Spark launching in local mode but not on yarn clusrer

No it didn't launch sample applications in cluster mode. I'll try running the script and will let you know.