I added an old server already used as datanode in my cluster. The process get completed, but as soon as the datanode is been added I get the following error message on cloudera manager: "The DataNode has 1 volume failure but disks"   I don't have any error message, just one warning as soon as I added the host:     5:27:37.578 PM WARN Storage Failed to add storage directory [DISK]file:/data/2/dfs/dn/ org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/2/dfs/dn is in an inconsistent state: Can't format the storage directory because the current/ directory is not empty. at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:495) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:600) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:279) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:418) at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1560) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1520) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:354) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) at java.lang.Thread.run(Thread.java:748)     and this are the common INFO log from that host:     5:40:11.726 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685 for deletion 5:40:11.727 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685 5:42:14.918 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874 src: /10.179.104.167:55132 dest: /10.179.104.168:50010 5:42:14.937 PM INFO clienttrace src: /10.179.104.167:55132, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1219924065_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, duration: 12451111 5:42:14.937 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 5:42:17.729 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687 for deletion 5:42:17.731 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687 5:43:14.919 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875 src: /10.179.104.167:55140 dest: /10.179.104.168:50010 5:43:14.938 PM INFO clienttrace src: /10.179.104.167:55140, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-879119697_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, duration: 12054137 5:43:14.938 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 5:43:17.732 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688 for deletion 5:43:17.733 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688 5:44:14.936 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876 src: /10.179.104.165:56144 dest: /10.179.104.168:50010 5:44:14.950 PM INFO clienttrace src: /10.179.104.165:56144, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_573364305_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, duration: 11591201 5:44:14.951 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:44:17.736 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689 for deletion 5:44:17.738 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689 5:46:14.937 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878 src: /10.179.104.165:56158 dest: /10.179.104.168:50010 5:46:14.951 PM INFO clienttrace src: /10.179.104.165:56158, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_562798393_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, duration: 11582892 5:46:14.951 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:46:17.744 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691 for deletion 5:46:17.746 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691 5:48:14.941 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880 src: /10.179.104.167:55180 dest: /10.179.104.168:50010 5:48:14.957 PM INFO clienttrace src: /10.179.104.167:55180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1107939178_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, duration: 13081545 5:48:14.958 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:48:20.748 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693 for deletion 5:48:20.750 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693 5:49:19.944 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881 src: /10.179.104.165:56180 dest: /10.179.104.168:50010 5:49:19.959 PM INFO clienttrace src: /10.179.104.165:56180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1732671963_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, duration: 12530220 5:49:19.960 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:49:26.752 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694 for deletion 5:49:26.754 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694       Any help on going deeply with this error? ... View more

Searching on internet I found that I can clean the tmp directory without problem on a cloudera cluster, so I did it. Now, when I try to start the cluster all services goes up well, but yarn failed to start with this error:   19/10/16 15:50:17 FATAL nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start. at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:79) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:329) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:563) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609) Caused by: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279) at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:65) ... 6 more Caused by: java.io.IOException: Unable to initialize WebAppContext at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:922) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274) ... 7 more Caused by: java.io.FileNotFoundException: /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp/webapps/node/.keep (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.mortbay.resource.JarResource.extract(JarResource.java:215) at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974) at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832) at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:900)   What can I do? ... View more