CDH5.0 VM: Error getting logs for job_1404657916663_0001

[#1]

Hello,

I am using CDH 5.0 Quickstart VM for VirtualBox. I run a mapreduce job and it completes successfully. The reduce code has several System.out.println statements which should be logged in the job log. But when I open logs in Hue>Jobbrowser, I get the error message:

Error getting logs for job_1404657916663_0001

When I see the Hue logs in /var/log/hadoop-mapreduce/hadoop-cmf-yarn-JOBHISTORY-localhost.localdomain.log.out, I find exceptions like the one below:

2014-07-06 08:18:20,767 ERROR org.apache.hadoop.yarn.webapp.View: Error getting logs for job_1404659730687_0001
org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=EXECUTE, inode="/tmp/logs/cloudera/logs":cloudera:supergroup:drwxrwx---
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:205)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:168)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5461)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5443)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:5405)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1680)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1632)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1612)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1586)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:482)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

So, I do:

hdfs dfs -chmod -R -777 /tmp/logs/cloudera

drwxrwxrwx   - cloudera supergroup          0 2014-07-06 08:23 /tmp/logs/cloudera/logs

Then I run the napreduce job again. I still get the same error in job browser.

The permissions to the log directory have reverted.

drwxrwx---   - cloudera supergroup          0 2014-07-06 08:23 /tmp/logs/cloudera/logs

What should be done to get the logs for the job?

Thanks in advance.

Lohith.

[#2]

Usually when I get this type of problem I reset the HDFS directory
structure like:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_y...

Romain

[#3]

Hi Romain,

Thanks for the reply.

I did a

sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/logs

When I ran jobs as 'Cloudera', no new logs were written to the above directory. In Hue, I got a message that no logs were available and aggregation may not have been complete.

Then I reverted as follows:

sudo -u hdfs hadoop fs -chown -R cloudera:supergroup /tmp/logs

Then, when the jobs ran, log directories were created in /tmp/logs/cloudera/logs. However, in Hue, I get the original error message.

The new directories were created as below:

drwxrwx---   - cloudera supergroup          0 2014-07-09 04:03 /tmp/logs/cloudera/logs/application_1404900446410_0005
-rw-r-----   1 cloudera supergroup    3755405 2014-07-09 04:03 /tmp/logs/cloudera/logs/application_1404900446410_0005/localhost.localdomain_8041

The permissions are not correct.

In the files in the tmp folder, I can see the log statements from my reduce program. But it is not available in Hue.

What should I do?

Thanks in advance!

[#4]

Hello,

I 'm having the same problem:

I tried: 

sudo -u hdfs hadoop fs -ls -R /tmp

 but this didn't worked...

The log files are written in the tmp folder, but hue is not finding them..

[#5 · Accepted solution]

Greetings, 

The proper perms are

/tmp/logs should be 777 and should have sticky bit set (the 1 in the 1777, this makes all child directories inherit the group of parent dir aka hadoop). 

sudo -u hdfs hdfs dfs -chmod 1777 /tmp/logs

sudo -u hdfs hdfs dfs -chown mapred:hadoop /tmp/logs

sudo -u hdfs hdfs dfs -chgrp -R hadoop /tmp/logs

After this I would restart your JobHistory server. 

[#6]

Thank you  mageru9 . This worked!!

The proper perms are

/tmp/logs should be 777 and should have sticky bit set (the 1 in the 1777, this makes all child directories inherit the group of parent dir aka hadoop). 

sudo -u hdfs hdfs dfs -chmod 1777 /tmp/logs

sudo -u hdfs hdfs dfs -chown mapred:hadoop /tmp/logs

sudo -u hdfs hdfs dfs -chgrp -R hadoop /tmp/logs

After this I would restart your JobHistory server. 

[#7]

 Was unable to re direct completed spark jobs from yarn to spark history server even though all permissions and spark conf was set correctly.

Might be useful.

The issue was we were passing a spark.conf file while submitting the spark job hoping the config changes would be aggregated with default parameters from default spark.conf.
Turns out it overrides the default spark config file. Even if you pass blank spark conf it will not consider the default spark.conf for the job.
We had to below 3 lines on the custom spark conf file to enable log aggregation at spark history server and URL at resource manager to point to spark history server.
This has to be done with every spark job. If a job is submitted with below 3 parms it will not be available in spark history server even if u restart anything.
```spark.eventLog.enabled=true
spark.eventLog.dir=hdfs://nameservice1/user/spark/applicationHistory
spark.yarn.historyServer.address=http://sparkhist-dev.visibleworld.com:18088```
https://community.cloudera.com/t5/CDH-Manual-Installation/Permission-denied-user-mapred-access-WRITE...

[#8]

We were unable to access Spark app log files from either YARN or Spark History Server UIs, with error "Error getting logs at <worker_node>:8041".  We can see the logs with "yarn logs" command.  Turns out our yarn.nodemanager.remote-app-log-dir = /tmp/logs, but the directory was owned by "yarn:yarn".  Following your instruction fixed the issue. 

Thanks a lot!

Miles

[#9]

Hi,

For me it's happening while finding YARN logs using yarn logs command. The /tmp/logs directory is having below permissions.

drwxrwxrwt - yarn yarn

But inside logs directory, the directories are having permission as below.

drwxrwxrwt - user/owner_of_directory yarn

Your guidance will be helpful.

Thanks,

[#10]

😁