Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

job history not available

avatar

When i submit jobs(from any one of the data nodes) within the cluster, i am able to see the job history. But when i submit a job from a remote client (it is not an edge node), the job gets completed. But the job history for the correspondign job is not available in the job history url. Can anyone help. IS there any particular setting that needs to be done on the client machine.

1 ACCEPTED SOLUTION

avatar

The issue got resolved and this is a brief of what is done.

As i said earlier, the node from where the jobs were submitted was not a part of the cluster and the client configs were missing certain parameters.

jsut added the following parameters to the mapred-site.xml

<property> <name>mapreduce.jobhistory.done-dir</name> <value>/mr-history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/mr-history/tmp</value> </property>

and it worked. Seems the job history server used to pick the job status from these directories and since these properties

were not mentioned in hte config files, the client couldnt write to the hdfs. so job history server just could not fetch them.

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@ARUNKUMAR RAMASAMY why wouldn't you make this node an edge node? You need clients on this node to see the whole picture.

avatar

hi @Artem Ervits. that node has all clients. but it is used for other purposes also. So it cannot be made as a dedicated edge node. but that node has apache hadoop, whereas the cluster is in HDP. will this be a cause for the issue?. Or will it be a network issue. any logs i should look for

avatar
Master Mentor

@ARUNKUMAR RAMASAMY take a look at our manual installation docs to set up logs.

avatar
Guru

If the job got completed, most likely its connected to a different cluster. You can check 'yarn log' on the cluster to see if the job got submitted to this cluster. Also, check from RM UI (RM:8080) to see if you can see the job in RM UI.

avatar
Master Mentor

@ARUNKUMAR RAMASAMY are you still having problems with this? Can you provide your own solution or accept best answer?

avatar

The issue got resolved and this is a brief of what is done.

As i said earlier, the node from where the jobs were submitted was not a part of the cluster and the client configs were missing certain parameters.

jsut added the following parameters to the mapred-site.xml

<property> <name>mapreduce.jobhistory.done-dir</name> <value>/mr-history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/mr-history/tmp</value> </property>

and it worked. Seems the job history server used to pick the job status from these directories and since these properties

were not mentioned in hte config files, the client couldnt write to the hdfs. so job history server just could not fetch them.