Reply
TS
Contributor
Posts: 84
Registered: ‎02-10-2015
Accepted Solution

[YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

I have created a new user (user2) and when running any MapReduce jobs the JobHistoryServer portal (port 19888) does NOT show any!!!

I can see the job from witihin the ResourceManager Web UI (port 8088) and when I click on the 'History' link (under the Tracking UI column) I am getting the error: "No Found: job_xxxxxxxxxxxxx_xxxx).

This happening for the MapReduce jobs only!!!

When I run Spark Jobs (under the same userid, user2) I am able to see the 'History' logs!!!

 

An existing userid (user1) works fine!

It seems to me that the issue is permissions-related.

 

Here are the HDFS permissions of the userids:

[root@master ~]# hdfs dfs -ls /tmp/logs
Found 5 items
drwxrwxrwt   - user1 hadoop              0 2015-04-20 08:48 /tmp/logs/user1
drwxrwxrwt   - hdfs    supergroup          0 2015-04-09 14:59 /tmp/logs/hdfs
drwxrwxrwx   - user2   hadoop              0 2015-05-06 16:23 /tmp/logs/user2
drwxrwxrwt   - root    hadoop              0 2015-04-09 16:46 /tmp/logs/root

 

Also, here are the persmissions of the HDFS /user/history/done and /user/history/done_intermediate folders:

[root@master ~]# hdfs dfs -ls /user/history
Found 2 items
drwxrwx---   - mapred hadoop          0 2015-05-11 10:32 /user/history/done
drwxrwxrwt   - mapred hadoop          0 2015-05-18 14:39 /user/history/done_intermediate

 

 

Cloudera Employee
Posts: 8
Registered: ‎07-08-2013

Re: [YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Can you browse /user/history/done_intermediate on HDFS to see if the files are copied? The files are copied there by the MapReduce AM and later read by JHS.

If the files aren't present there, the MapReduce AM is likely not copying them. Can we make sure mapreduce.jobhistory.intermediate-done-dir is set to the same value in the AM and JHS? If not set, it uses yarn.app.mapreduce.am.staging-dir. We should check if that value is the same between the AM and JHS.
Karthik Kambatla
Software Engineer, Cloudera Inc.
TS
Contributor
Posts: 84
Registered: ‎02-10-2015

Re: [YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Thank you for your comment!

The issue has been resolved, it had to do with permissions!

I had to reset the mode of the /user/history/done folder /user/history/done_intermediate was good!

 

Highlighted
New Contributor
Posts: 4
Registered: ‎06-05-2017

Re: [YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Hi TS,

 

     What kind of permissions do you have to set on the /user/history/done folder to resolve the issue?

 

Best,

SKA

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.