Resolving YARN JHS Error "Not Found: job_" by Fixing HDFS /user/history/done Permissions

[#1]

I have created a new user (user2) and when running any MapReduce jobs the JobHistoryServer portal (port 19888) does NOT show any!!!

I can see the job from witihin the ResourceManager Web UI (port 8088) and when I click on the 'History' link (under the Tracking UI column) I am getting the error: "No Found: job_xxxxxxxxxxxxx_xxxx).

This happening for the MapReduce jobs only!!!

When I run Spark Jobs (under the same userid, user2) I am able to see the 'History' logs!!!

An existing userid (user1) works fine!

It seems to me that the issue is permissions-related.

Here are the HDFS permissions of the userids:

[root@master ~]# hdfs dfs -ls /tmp/logs
Found 5 items
drwxrwxrwt   - user1 hadoop              0 2015-04-20 08:48 /tmp/logs/user1
drwxrwxrwt   - hdfs    supergroup          0 2015-04-09 14:59 /tmp/logs/hdfs
drwxrwxrwx   - user2   hadoop              0 2015-05-06 16:23 /tmp/logs/user2
drwxrwxrwt   - root    hadoop              0 2015-04-09 16:46 /tmp/logs/root

Also, here are the persmissions of the HDFS /user/history/done and /user/history/done_intermediate folders:

[root@master ~]# hdfs dfs -ls /user/history
Found 2 items
drwxrwx---   - mapred hadoop          0 2015-05-11 10:32 /user/history/done
drwxrwxrwt   - mapred hadoop          0 2015-05-18 14:39 /user/history/done_intermediate

[#2]

Can you browse /user/history/done_intermediate on HDFS to see if the files are copied? The files are copied there by the MapReduce AM and later read by JHS.

If the files aren't present there, the MapReduce AM is likely not copying them. Can we make sure mapreduce.jobhistory.intermediate-done-dir is set to the same value in the AM and JHS? If not set, it uses yarn.app.mapreduce.am.staging-dir. We should check if that value is the same between the AM and JHS.

[#3 · Accepted solution]

Thank you for your comment!

The issue has been resolved, it had to do with permissions!

I had to reset the mode of the /user/history/done folder /user/history/done_intermediate was good!

[#4]

Hi TS,

     What kind of permissions do you have to set on the /user/history/done folder to resolve the issue?

Best,

SKA