Error creating done directory: [file:/user/history/done]

[#1]

2024-05-31 19:23:17,927 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServer
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [file:/user/history/done]
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:701)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:637)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:598)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:97)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:150)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:226)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:236)
Caused by: java.io.IOException: mkdir of file:/user/history/done failed
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1320)
at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:185)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:214)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:809)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:805)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:812)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.mkdir(HistoryFileManager.java:756)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:684)
... 10 more
2024-05-31 19:23:17,930 INFO org.apache.hadoop.util.ExitUtil: Exiting with status -1: Error starting JobHistoryServer



yaya9800-1# ll /ifs/hdfs/hadoop-root
total 40
drwxr-xr-x 2 hbase hbase 0 May 31 11:10 hbase
drwxrwxr-x 2 solr solr 0 May 31 11:10 solr
drwxrwxrwt 4 hdfs supergroup 44 May 31 11:10 tmp
drwxr-xr-x 12 hdfs supergroup 229 May 31 11:10 user
yaya9800-1# ll /ifs/hdfs/hadoop-root/user
total 20
drwxrwxr-x 2 flume flume 0 May 31 11:10 flume
drwxr-xr-x 2 hdfs hdfs 0 May 31 11:10 hdfs
drwxrwxrwx 2 mapred hadoop 0 May 31 11:10 history
drwxrwxr-x 3 hive hive 27 May 31 11:10 hive
drwxr-xr-x 3 hue hue 57 May 31 11:10 hue
drwxrwxr-x 2 impala impala 0 May 31 11:10 impala
drwxrwxr-x 2 oozie oozie 0 May 31 11:10 oozie
drwxr-x--x 3 spark spark 36 May 31 11:10 spark
drwxrwxr-x 2 sqoop2 sqoop 0 May 31 11:10 sqoop2
drwxr-xr-x 2 yarn yarn 0 May 31 11:10 yarn
yaya9800-1# ll /ifs/hdfs/hadoop-root/user/history
total 0
yaya9800-1#

[#2 · Accepted solution]

Hello @vec

It seems like it is trying to use a local file system instead of HDFS.

Can you check what the value is set for hdfs_service in CM -> YARN -> Configuration > hdfs_service ?

Make sure it is set to use "hdfs" 

[#3 · Accepted solution]

@vec  file:/ represents the local file system.

Caused by: java.io.IOException: mkdir of file:/user/history/done failed

[#4 · Accepted solution]

Hello @vec

It seems like it is trying to use a local file system instead of HDFS.

Can you check what the value is set for hdfs_service in CM -> YARN -> Configuration > hdfs_service ?

Make sure it is set to use "hdfs" 

[#5]

After changing to powerscale hdfs in YARN , then it works. Thank you @Scharan 

[#6]

@Scharan how do you tell it is using local file system in the log ?

[#7 · Accepted solution]

@vec  file:/ represents the local file system.

Caused by: java.io.IOException: mkdir of file:/user/history/done failed

[#8 · Accepted solution]

Hello @vec 

If you found  the suggestions/solutions provided helped you with solving your issue  please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,