Support Questions

Find answers, ask questions, and share your expertise

Yarn commands not reading resource manager hostname from

New Contributor

I am trying to setup spark on yarn cluster using Hadoop cookbook using HDP distribution. As part of this i am using to configure yarn resourcemanager.

I have the following in

export YARN_RESOURCEMANAGER_OPTS="-Dyarn.resourcemanager.hostname="

I am able to see the cluster on the This also shows 2 nodemanagers connected to it. But when i run

yarn node --list

it tries to connect to and gives the following log

18/04/30 06:09:29 INFO client.RMProxy: Connecting to ResourceManager at /

18/04/30 06:09:31 INFO ipc.Client: Retrying connect to server: Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

After some retries it fails.

The same error is received when i run spark-submit using following command

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --executor-memory 1G /usr/hdp/

as spark uses $HADOOP_CONF_DIR for getting yarn configurations.

What is the cause. Does yarn command reads only from yarn-site.xml and not


Rising Star is used when you run any yarn command. So it works if you use the yarn command to submit a mapreduce job as below.

yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 5 5

But spark-submit command doesn't invoke, so it will read the yarn-site.xml from $HADOOP_CONF_DIR and gets resourcemanager address from it.

New Contributor

@Tarun Parimi But yarn commands are also trying to connect to Instead of resource manager IP Address. This happens on both master and slave machines.

Rising Star

I didn't notice that you were only setting YARN_RESOURCEMANAGER_OPTS. This env variable is used for only the resourcemanger daemon. So to specify the opts for all hadoop and yarn client commands, you can use HADOOP_CLIENT_OPTS in . .

export HADOOP_CLIENT_OPTS="-Dyarn.resourcemanager.hostname="

But I am not sure why you would need to this when you can just set it in the yarn-site.xml, which is what is recommended.