Job submitted on edge node runs in local host and cannot be tracked under resource manager web ui

[#1]

Hi,

I have 8 node cluster, when i submit job in edge node (Pi program), it creates job in local and executes

hadoop jar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/jars/hadoop-examples.jar pi 10 10
Number of Maps = 10
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
17/11/20 08:47:57 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/11/20 08:47:57 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/11/20 08:47:58 INFO input.FileInputFormat: Total input paths to process : 10
17/11/20 08:47:58 INFO mapreduce.JobSubmitter: number of splits:10
17/11/20 08:47:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local635221628_0001
17/11/20 08:47:58 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/11/20 08:47:58 INFO mapreduce.Job: Running job: job_local635221628_0001
17/11/20 08:47:58 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/11/20 08:47:58 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/20 08:47:58 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/11/20 08:47:58 INFO mapred.LocalJobRunner: Waiting for map tasks
17/11/20 08:47:58 INFO mapred.LocalJobRunner: Starting task: attempt_local635221628_0001_m_000000_0
17/11/20 08:47:58 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/11/20 08:47:58 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/11/20 08:47:58 INFO mapred.MapTask: Processing split: hdfs://nameservice-ha/user/hduser/QuasiMonteCarlo_1511185676373_1845096796/in/part0:0+118

but it executes succssfully.. But the job id job_localxxx cannot be tracked under Resource manager web ui.

When I run the same job on any other node (Name node or worker node, proper job_id is getting created which will be available in resoure manager web ui)

Also I noticed, when I run mapred job -list in edge node, throws me below error

mapred job -list
17/11/20 08:52:30 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/11/20 08:52:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.mapreduce.tools.CLI.listJobs(CLI.java:604)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:382)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1269)

And when I run 

yarn application -list
17/11/20 08:52:59 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/11/20 08:53:00 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/11/20 08:53:01 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/11/20 08:53:02 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/11/20 08:53:03 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Where as these commands works fine in other nodes, I have oozie service installed and ResourceManager Address is set to 8032. 

Can some one tell me what went wrong? How can I fix this issue? 

[#2 · Accepted solution]

The default behaviour of Hadoop is to run things locally, in face of no
found YARN cluster configuration. In CM managed clusters, cluster
configuration for client programs are deployed by means of a Gateway role.
Your edge host is missing a gateway role and the subsequent config files
required to discover and use the cluster daemons.

Do these two steps:

1. Visit YARN -> Instances page in CM, then click 'Add Role Instances' and
under the Gateway type in the modal dialog, find and add your edge hostname
to it (this edge host should already be running a CM agent for it to show
up here).

2. Deploy cluster-wide client configs, following this:
https://www.youtube.com/watch?v=4S9H3wftM_0

Retry your commands after this completes. Also verify that your edge host
now has a proper /etc/hadoop/conf symlink, with the directory contents
carrying info about the cluster.

P.s. Having HDFS Gateways is insufficient to connect to YARN, you will need
a YARN Gateway to connect to YARN.

[#3]

Thanks a lot.. This resolved the issue : ) 

I have one more doubt, If I get java heap size issue like,

Caused by: java.lang.OutOfMemoryError: Java heap space when running any mapreduce job, how to increase the java heap size runtime? Does “-Dmapreduce.map.java.opts=-Xmx2048m” this really do something there? I dint find any changes. Could you please advice the best way to increase java heap size? Thanks in advance