Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

job history not available

Solved Go to solution

job history not available

When i submit jobs(from any one of the data nodes) within the cluster, i am able to see the job history. But when i submit a job from a remote client (it is not an edge node), the job gets completed. But the job history for the correspondign job is not available in the job history url. Can anyone help. IS there any particular setting that needs to be done on the client machine.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: job history not available

The issue got resolved and this is a brief of what is done.

As i said earlier, the node from where the jobs were submitted was not a part of the cluster and the client configs were missing certain parameters.

jsut added the following parameters to the mapred-site.xml

<property> <name>mapreduce.jobhistory.done-dir</name> <value>/mr-history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/mr-history/tmp</value> </property>

and it worked. Seems the job history server used to pick the job status from these directories and since these properties

were not mentioned in hte config files, the client couldnt write to the hdfs. so job history server just could not fetch them.

6 REPLIES 6

Re: job history not available

Mentor

@ARUNKUMAR RAMASAMY why wouldn't you make this node an edge node? You need clients on this node to see the whole picture.

Re: job history not available

hi @Artem Ervits. that node has all clients. but it is used for other purposes also. So it cannot be made as a dedicated edge node. but that node has apache hadoop, whereas the cluster is in HDP. will this be a cause for the issue?. Or will it be a network issue. any logs i should look for

Re: job history not available

Mentor

@ARUNKUMAR RAMASAMY take a look at our manual installation docs to set up logs.

Re: job history not available

Guru

If the job got completed, most likely its connected to a different cluster. You can check 'yarn log' on the cluster to see if the job got submitted to this cluster. Also, check from RM UI (RM:8080) to see if you can see the job in RM UI.

Re: job history not available

Mentor

@ARUNKUMAR RAMASAMY are you still having problems with this? Can you provide your own solution or accept best answer?

Re: job history not available

The issue got resolved and this is a brief of what is done.

As i said earlier, the node from where the jobs were submitted was not a part of the cluster and the client configs were missing certain parameters.

jsut added the following parameters to the mapred-site.xml

<property> <name>mapreduce.jobhistory.done-dir</name> <value>/mr-history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/mr-history/tmp</value> </property>

and it worked. Seems the job history server used to pick the job status from these directories and since these properties

were not mentioned in hte config files, the client couldnt write to the hdfs. so job history server just could not fetch them.