Support Questions
Find answers, ask questions, and share your expertise

Please advise -Getting the below error for spark history server in CDP7.1.3 cluster setup - java.io.FileNotFoundException: Log directory specified does not exist: hdfs://nameservice1/user/spark/applicationHistory

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:294)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: hdfs://nameservice1/user/spark/applicationHistory
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:268)
at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:212)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:208)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:87)
... 6 more
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/spark/applicationHistory
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1735)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1728)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1743)
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:258)
... 9 more

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Please advise -Getting the below error for spark history server in CDP7.1.3 cluster setup - java.io.FileNotFoundException: Log directory specified does not exist: hdfs://nameservice1/user/spark/applicationHistory

@vijaypabothu You can resolve this using below steps. This seems somehow the directory is not exists which is causing the issue. 

# hdfs dfs -mkdir -p /user/spark/applicationHistory
// make sure it is owned by spark user/group
# hdfs dfs -chown -R spark:spark /user/spark/
// set 777 access rights and sticky bit
# hdfs dfs -chmod 1777 /user/spark/applicationHistory
// restart the SHS process to test. CM > Spark > Instances > (select Spark History Server) > Actions > Restart

 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

2 REPLIES 2

Re: Please advise -Getting the below error for spark history server in CDP7.1.3 cluster setup - java.io.FileNotFoundException: Log directory specified does not exist: hdfs://nameservice1/user/spark/applicationHistory

@vijaypabothu You can resolve this using below steps. This seems somehow the directory is not exists which is causing the issue. 

# hdfs dfs -mkdir -p /user/spark/applicationHistory
// make sure it is owned by spark user/group
# hdfs dfs -chown -R spark:spark /user/spark/
// set 777 access rights and sticky bit
# hdfs dfs -chmod 1777 /user/spark/applicationHistory
// restart the SHS process to test. CM > Spark > Instances > (select Spark History Server) > Actions > Restart

 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

Re: Please advise -Getting the below error for spark history server in CDP7.1.3 cluster setup - java.io.FileNotFoundException: Log directory specified does not exist: hdfs://nameservice1/user/spark/applicationHistory

Thanks a lot @GangWar !! Its working.