why namenode looking on file under spark2-history

mike_bronson7 · ‎06-12-2018

when we start the first namenode , we get "ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode. java.io.FileNotFoundException: File does not exist: /spark2-history/.bbf98e62-edf0-4758-9fb7-5e4caf93e32e"

why namenode is looking on the non exisinting file under /spark2-history ?

2018-06-12 05:26:49,313 WARN  namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(705)) - Encountered exception loading fsimage
java.io.FileNotFoundException: File does not exist: /spark2-history/.bbf98e62-edf0-4758-9fb7-5e4caf93e32e
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:433)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:703)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769)
2018-06-12 05:26:49,315 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@master01.sys764.com:50070
2018-06-12 05:26:49,416 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping NameNode metrics system...
2018-06-12 05:26:49,417 INFO  impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.
2018-06-12 05:26:49,418 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NameNode metrics system stopped.
2018-06-12 05:26:49,419 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - NameNode metrics system shutdown complete.
2018-06-12 05:26:49,419 ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode.
java.io.FileNotFoundException: File does not exist: /spark2-history/.bbf98e62-edf0-4758-9fb7-5e4caf93e32e
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:433)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:703)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769)
2018-06-12 05:26:49,421 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2018-06-12 05:26:49,487 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:

Michael-Bronson

falbani · ‎06-12-2018

@Michael Bronson The above error could potentially be caused by Spark History Server which usually reads from /spark2-history folder or a running spark application. Perhaps NN was restarted and Spark History Server was still running or a spark application was running during this time.

HTH