Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Solved Go to solution
Highlighted

[YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Rising Star

I have created a new user (user2) and when running any MapReduce jobs the JobHistoryServer portal (port 19888) does NOT show any!!!

I can see the job from witihin the ResourceManager Web UI (port 8088) and when I click on the 'History' link (under the Tracking UI column) I am getting the error: "No Found: job_xxxxxxxxxxxxx_xxxx).

This happening for the MapReduce jobs only!!!

When I run Spark Jobs (under the same userid, user2) I am able to see the 'History' logs!!!

 

An existing userid (user1) works fine!

It seems to me that the issue is permissions-related.

 

Here are the HDFS permissions of the userids:

[root@master ~]# hdfs dfs -ls /tmp/logs
Found 5 items
drwxrwxrwt   - user1 hadoop              0 2015-04-20 08:48 /tmp/logs/user1
drwxrwxrwt   - hdfs    supergroup          0 2015-04-09 14:59 /tmp/logs/hdfs
drwxrwxrwx   - user2   hadoop              0 2015-05-06 16:23 /tmp/logs/user2
drwxrwxrwt   - root    hadoop              0 2015-04-09 16:46 /tmp/logs/root

 

Also, here are the persmissions of the HDFS /user/history/done and /user/history/done_intermediate folders:

[root@master ~]# hdfs dfs -ls /user/history
Found 2 items
drwxrwx---   - mapred hadoop          0 2015-05-11 10:32 /user/history/done
drwxrwxrwt   - mapred hadoop          0 2015-05-18 14:39 /user/history/done_intermediate

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: [YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Rising Star

Thank you for your comment!

The issue has been resolved, it had to do with permissions!

I had to reset the mode of the /user/history/done folder /user/history/done_intermediate was good!

 

3 REPLIES 3

Re: [YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Cloudera Employee
Can you browse /user/history/done_intermediate on HDFS to see if the files are copied? The files are copied there by the MapReduce AM and later read by JHS.

If the files aren't present there, the MapReduce AM is likely not copying them. Can we make sure mapreduce.jobhistory.intermediate-done-dir is set to the same value in the AM and JHS? If not set, it uses yarn.app.mapreduce.am.staging-dir. We should check if that value is the same between the AM and JHS.
Karthik Kambatla
Software Engineer, Cloudera Inc.

Re: [YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

Rising Star

Thank you for your comment!

The issue has been resolved, it had to do with permissions!

I had to reset the mode of the /user/history/done folder /user/history/done_intermediate was good!

 

Re: [YARN] JHS M/R Job not Found (Not Found: job_xxxxxxxxxxxxxxx_xxxx)

New Contributor

Hi TS,

 

     What kind of permissions do you have to set on the /user/history/done folder to resolve the issue?

 

Best,

SKA

Don't have an account?
Coming from Hortonworks? Activate your account here