YARN(MR2 Included) JobHistory Server failing to start in Cluster

[#1]

Hi

YARN keeps failing with the error below:

Permission denied. user=mapred is not the owner of inode=/user/history/done

 The folder does exists and here's the ownership:

drwxrwxrwx - cloudera-scm hdfs 0 2017-12-01 09:29 /user/history/done/2017
drwxrwxrwx - cloudera-scm hdfs 0 2018-11-01 07:20 /user/history/done/2018

Context: we just upgraded from Cloudera/CDH from 5/12 to 6.2

Exception Snippet

Service org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://gislny10.elab.fictcorp.com:8020/user/history/done]
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://gislny10.elab.fictcorp.com:8020/user/history/done]
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:696)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:630)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:591)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:97)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:150)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:226)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:236)
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=mapred is not the owner of inode=/user/history/done
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:270)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1855)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1839)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1784)
at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:64)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1861)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:856)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:509)

[#2]

@ChineduLB,

You have showed the permission and ownership of sub-directories for /user/history/done:

drwxrwxrwx - cloudera-scm hdfs 0 2017-12-01 09:29 /user/history/done/2017
drwxrwxrwx - cloudera-scm hdfs 0 2018-11-01 07:20 /user/history/done/2018

What about the ownership of /user/history/done itself? Can you run below command share the output:

hdfs dfs -ls -d /user/history
hdfs dfs -ls -d /user/history/done

Cheers
Eric

[#3]

hdfs dfs -ls -d /user/history

  drwxrwxrwx   - hdfs hdfs          0 2017-07-21 02:41 /user/history

hdfs dfs -ls -d /user/history/done

  drwxrwxrwx   - hdfs hdfs          0 2018-01-09 12:27 /user/history/done

[#4]

@ChineduLB ,

Thanks for the update.

/user/history and /user/history/done need to be owned by mapred:hadoop, see below:

drwxrwxrwx - mapred hadoop 0 2017-12-11 06:23 /user/history
drwxrwx--- - mapred hadoop 0 2019-01-15 04:31 /user/history/done
drwxrwx--- - mapred hadoop 0 2018-12-03 11:30 /user/history/done/2018
drwxrwx--- - mapred hadoop 0 2019-09-10 05:46 /user/history/done/2019

Please update accordingly and test again.

Cheers

[#5]

Having more problems. struggling with Unix.

issued command:

hdfs dfs -chown -R  mapred:hdfs /user/history/

got Error:

chown: changing ownership of '/user/history': Permission denied. user=root is not the owner of inode=/user/history

[#6]

@ChineduLB

You need to be running the command using "hdfs" user, which is a superuser in HDFS. Please try again with that.

Cheers
Eric

[#7]

How can I find the password for the hdfs user.

last time I looked it was all asterix in the box in CM.  is there a way to get the actual text of the hdfs user

PS: The guys that setup the cluster lost all details.

I'm trying to recover it now. Just migrated from 5.12 to 6.2 and been facing one issue after another

regards,

[#8]

Try to update password using kadmin.local after you log into the KDC server, reference here:
http://web.mit.edu/KERBEROS/krb5-1.4/krb5-1.4.1/doc/krb5-admin/Changing-Passwords.html

Cheers