Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

YARN(MR2 Included) JobHistory Server failing to start in Cluster

YARN(MR2 Included) JobHistory Server failing to start in Cluster

Explorer

Hi

YARN keeps failing with the error below:

Permission denied. user=mapred is not the owner of inode=/user/history/done

 

 The folder does exists and here's the ownership:

drwxrwxrwx - cloudera-scm hdfs 0 2017-12-01 09:29 /user/history/done/2017
drwxrwxrwx - cloudera-scm hdfs 0 2018-11-01 07:20 /user/history/done/2018

 

Context: we just upgraded from Cloudera/CDH from 5/12 to 6.2

 

Exception Snippet

Service org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://gislny10.elab.fictcorp.com:8020/user/history/done]
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://gislny10.elab.fictcorp.com:8020/user/history/done]
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:696)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:630)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:591)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:97)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:150)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:226)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:236)
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=mapred is not the owner of inode=/user/history/done
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:270)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1855)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1839)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1784)
at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:64)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1861)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:856)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:509)

7 REPLIES 7

Re: YARN(MR2 Included) JobHistory Server failing to start in Cluster

Guru
@ChineduLB,

You have showed the permission and ownership of sub-directories for /user/history/done:

drwxrwxrwx - cloudera-scm hdfs 0 2017-12-01 09:29 /user/history/done/2017
drwxrwxrwx - cloudera-scm hdfs 0 2018-11-01 07:20 /user/history/done/2018

What about the ownership of /user/history/done itself? Can you run below command share the output:

hdfs dfs -ls -d /user/history
hdfs dfs -ls -d /user/history/done

Cheers
Eric

Re: YARN(MR2 Included) JobHistory Server failing to start in Cluster

Explorer

 

hdfs dfs -ls -d /user/history

  drwxrwxrwx   - hdfs hdfs          0 2017-07-21 02:41 /user/history


hdfs dfs -ls -d /user/history/done

  drwxrwxrwx   - hdfs hdfs          0 2018-01-09 12:27 /user/history/done

Highlighted

Re: YARN(MR2 Included) JobHistory Server failing to start in Cluster

Guru
@ChineduLB ,

Thanks for the update.

/user/history and /user/history/done need to be owned by mapred:hadoop, see below:

drwxrwxrwx - mapred hadoop 0 2017-12-11 06:23 /user/history
drwxrwx--- - mapred hadoop 0 2019-01-15 04:31 /user/history/done
drwxrwx--- - mapred hadoop 0 2018-12-03 11:30 /user/history/done/2018
drwxrwx--- - mapred hadoop 0 2019-09-10 05:46 /user/history/done/2019

Please update accordingly and test again.

Cheers

Re: YARN(MR2 Included) JobHistory Server failing to start in Cluster

Explorer

Having more problems. struggling with Unix.

 

issued command:

hdfs dfs -chown -R  mapred:hdfs /user/history/

 

got Error:

chown: changing ownership of '/user/history': Permission denied. user=root is not the owner of inode=/user/history

Re: YARN(MR2 Included) JobHistory Server failing to start in Cluster

Guru
@ChineduLB

You need to be running the command using "hdfs" user, which is a superuser in HDFS. Please try again with that.

Cheers
Eric

Re: YARN(MR2 Included) JobHistory Server failing to start in Cluster

Explorer

How can I find the password for the hdfs user.

last time I looked it was all asterix in the box in CM.  is there a way to get the actual text of the hdfs user

 

PS: The guys that setup the cluster lost all details.

I'm trying to recover it now. Just migrated from 5.12 to 6.2 and been facing one issue after another

 

regards,

 

Re: YARN(MR2 Included) JobHistory Server failing to start in Cluster

Guru
Try to update password using kadmin.local after you log into the KDC server, reference here:
http://web.mit.edu/KERBEROS/krb5-1.4/krb5-1.4.1/doc/krb5-admin/Changing-Passwords.html

Cheers