Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

why namenode looking on file under spark2-history

when we start the first namenode , we get "ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode. java.io.FileNotFoundException: File does not exist: /spark2-history/.bbf98e62-edf0-4758-9fb7-5e4caf93e32e"

why namenode is looking on the non exisinting file under /spark2-history ?

2018-06-12 05:26:49,313 WARN  namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(705)) - Encountered exception loading fsimage
java.io.FileNotFoundException: File does not exist: /spark2-history/.bbf98e62-edf0-4758-9fb7-5e4caf93e32e
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:433)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:703)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769)
2018-06-12 05:26:49,315 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@master01.sys764.com:50070
2018-06-12 05:26:49,416 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping NameNode metrics system...
2018-06-12 05:26:49,417 INFO  impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.
2018-06-12 05:26:49,418 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NameNode metrics system stopped.
2018-06-12 05:26:49,419 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - NameNode metrics system shutdown complete.
2018-06-12 05:26:49,419 ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode.
java.io.FileNotFoundException: File does not exist: /spark2-history/.bbf98e62-edf0-4758-9fb7-5e4caf93e32e
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:433)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:703)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769)
2018-06-12 05:26:49,421 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2018-06-12 05:26:49,487 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
Michael-Bronson
1 REPLY 1

@Michael Bronson The above error could potentially be caused by Spark History Server which usually reads from /spark2-history folder or a running spark application. Perhaps NN was restarted and Spark History Server was still running or a spark application was running during this time.

HTH