job history not available

[#1]

When i submit jobs(from any one of the data nodes) within the cluster, i am able to see the job history. But when i submit a job from a remote client (it is not an edge node), the job gets completed. But the job history for the correspondign job is not available in the job history url. Can anyone help. IS there any particular setting that needs to be done on the client machine.

[#2]

@ARUNKUMAR RAMASAMY why wouldn't you make this node an edge node? You need clients on this node to see the whole picture.

[#3]

hi @Artem Ervits. that node has all clients. but it is used for other purposes also. So it cannot be made as a dedicated edge node. but that node has apache hadoop, whereas the cluster is in HDP. will this be a cause for the issue?. Or will it be a network issue. any logs i should look for

[#4]

@ARUNKUMAR RAMASAMY take a look at our manual installation docs to set up logs.

[#5]

If the job got completed, most likely its connected to a different cluster. You can check 'yarn log' on the cluster to see if the job got submitted to this cluster. Also, check from RM UI (RM:8080) to see if you can see the job in RM UI.

[#6]

@ARUNKUMAR RAMASAMY are you still having problems with this? Can you provide your own solution or accept best answer?

[#7 · Accepted solution]

The issue got resolved and this is a brief of what is done.

As i said earlier, the node from where the jobs were submitted was not a part of the cluster and the client configs were missing certain parameters.

jsut added the following parameters to the mapred-site.xml

<property> <name>mapreduce.jobhistory.done-dir</name> <value>/mr-history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/mr-history/tmp</value> </property>

and it worked. Seems the job history server used to pick the job status from these directories and since these properties

were not mentioned in hte config files, the client couldnt write to the hdfs. so job history server just could not fetch them.