Support Questions

Find answers, ask questions, and share your expertise

The DataNode has 1 volume failure but disks seem fine

avatar
Explorer

I added an old server already used as datanode in my cluster.

The process get completed, but as soon as the datanode is been added I get the following error message on cloudera manager:

"The DataNode has 1 volume failure but disks"

 

I don't have any error message, just one warning as soon as I added the host:

 

 

5:27:37.578 PM	WARN	Storage	
Failed to add storage directory [DISK]file:/data/2/dfs/dn/
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/2/dfs/dn is in an inconsistent state: Can't format the storage directory because the current/ directory is not empty.
	at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:495)
	at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:600)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:279)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:418)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1560)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1520)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:354)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
	at java.lang.Thread.run(Thread.java:748)

 

 

and this are the common INFO log from that host:

 

 

5:40:11.726 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685 for deletion
5:40:11.727 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685
5:42:14.918 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874 src: /10.179.104.167:55132 dest: /10.179.104.168:50010
5:42:14.937 PM	INFO	clienttrace	
src: /10.179.104.167:55132, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1219924065_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, duration: 12451111
5:42:14.937 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
5:42:17.729 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687 for deletion
5:42:17.731 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687
5:43:14.919 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875 src: /10.179.104.167:55140 dest: /10.179.104.168:50010
5:43:14.938 PM	INFO	clienttrace	
src: /10.179.104.167:55140, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-879119697_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, duration: 12054137
5:43:14.938 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
5:43:17.732 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688 for deletion
5:43:17.733 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688
5:44:14.936 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876 src: /10.179.104.165:56144 dest: /10.179.104.168:50010
5:44:14.950 PM	INFO	clienttrace	
src: /10.179.104.165:56144, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_573364305_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, duration: 11591201
5:44:14.951 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:44:17.736 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689 for deletion
5:44:17.738 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689
5:46:14.937 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878 src: /10.179.104.165:56158 dest: /10.179.104.168:50010
5:46:14.951 PM	INFO	clienttrace	
src: /10.179.104.165:56158, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_562798393_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, duration: 11582892
5:46:14.951 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:46:17.744 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691 for deletion
5:46:17.746 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691
5:48:14.941 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880 src: /10.179.104.167:55180 dest: /10.179.104.168:50010
5:48:14.957 PM	INFO	clienttrace	
src: /10.179.104.167:55180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1107939178_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, duration: 13081545
5:48:14.958 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:48:20.748 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693 for deletion
5:48:20.750 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693
5:49:19.944 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881 src: /10.179.104.165:56180 dest: /10.179.104.168:50010
5:49:19.959 PM	INFO	clienttrace	
src: /10.179.104.165:56180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1732671963_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, duration: 12530220
5:49:19.960 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:49:26.752 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694 for deletion
5:49:26.754 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694

 

 

 

Any help on going deeply with this error?

1 ACCEPTED SOLUTION

avatar
Explorer

I moved the content on another directory, restarted the namenode and the error message is gone.

So I could delete the old directory

View solution in original post

4 REPLIES 4

avatar
Super Guru

First thing I see is:

 

Can't format the storage directory because the current/ directory is not empty.

 

Is the path/folder empty?    look for hidden files too...

avatar
Explorer

It's not empty. 

Is it safe to delete the content?

avatar
Super Guru

Not sure, it is your folder/files.

 

If you are not sure yourself,  just make another folder inside of that folder and try to use it.   

 

When you setup the data node to an empty folder it should resolve your issue.

avatar
Explorer

I moved the content on another directory, restarted the namenode and the error message is gone.

So I could delete the old directory