Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

The DataNode has 1 volume failure but disks seem fine

Labels:
avatar
author-rank Cico
Explorer

Created on ‎11-19-2019 08:58 AM - last edited on ‎11-19-2019 04:33 PM by Moderator

I added an old server already used as datanode in my cluster.

The process get completed, but as soon as the datanode is been added I get the following error message on cloudera manager:

"The DataNode has 1 volume failure but disks"

 

I don't have any error message, just one warning as soon as I added the host:

 

 

5:27:37.578 PM	WARN	Storage	
Failed to add storage directory [DISK]file:/data/2/dfs/dn/
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/2/dfs/dn is in an inconsistent state: Can't format the storage directory because the current/ directory is not empty.
	at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:495)
	at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:600)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:279)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:418)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1560)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1520)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:354)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
	at java.lang.Thread.run(Thread.java:748)

 

 

and this are the common INFO log from that host:

 

 

5:40:11.726 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685 for deletion
5:40:11.727 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685
5:42:14.918 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874 src: /10.179.104.167:55132 dest: /10.179.104.168:50010
5:42:14.937 PM	INFO	clienttrace	
src: /10.179.104.167:55132, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1219924065_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, duration: 12451111
5:42:14.937 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
5:42:17.729 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687 for deletion
5:42:17.731 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687
5:43:14.919 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875 src: /10.179.104.167:55140 dest: /10.179.104.168:50010
5:43:14.938 PM	INFO	clienttrace	
src: /10.179.104.167:55140, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-879119697_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, duration: 12054137
5:43:14.938 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
5:43:17.732 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688 for deletion
5:43:17.733 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688
5:44:14.936 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876 src: /10.179.104.165:56144 dest: /10.179.104.168:50010
5:44:14.950 PM	INFO	clienttrace	
src: /10.179.104.165:56144, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_573364305_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, duration: 11591201
5:44:14.951 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:44:17.736 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689 for deletion
5:44:17.738 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689
5:46:14.937 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878 src: /10.179.104.165:56158 dest: /10.179.104.168:50010
5:46:14.951 PM	INFO	clienttrace	
src: /10.179.104.165:56158, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_562798393_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, duration: 11582892
5:46:14.951 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:46:17.744 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691 for deletion
5:46:17.746 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691
5:48:14.941 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880 src: /10.179.104.167:55180 dest: /10.179.104.168:50010
5:48:14.957 PM	INFO	clienttrace	
src: /10.179.104.167:55180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1107939178_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, duration: 13081545
5:48:14.958 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:48:20.748 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693 for deletion
5:48:20.750 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693
5:49:19.944 PM	INFO	DataNode	
Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881 src: /10.179.104.165:56180 dest: /10.179.104.168:50010
5:49:19.959 PM	INFO	clienttrace	
src: /10.179.104.165:56180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1732671963_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, duration: 12530220
5:49:19.960 PM	INFO	DataNode	
PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
5:49:26.752 PM	INFO	FsDatasetAsyncDiskService	
Scheduling blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694 for deletion
5:49:26.754 PM	INFO	FsDatasetAsyncDiskService	
Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694

 

 

 

Any help on going deeply with this error?

4,439 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank Cico
Explorer

Created ‎11-20-2019 02:08 AM

I moved the content on another directory, restarted the namenode and the error message is gone.

So I could delete the old directory

View solution in original post

4,399 Views
0 Kudos
4 REPLIES 4

avatar
author-rank stevenmatison
Super Guru

Created ‎11-19-2019 10:14 AM

First thing I see is:

 

Can't format the storage directory because the current/ directory is not empty.

 

Is the path/folder empty?    look for hidden files too...

4,429 Views
0 Kudos

avatar
author-rank Cico
Explorer

Created ‎11-19-2019 10:17 AM

It's not empty. 

Is it safe to delete the content?

4,425 Views
0 Kudos

avatar
author-rank stevenmatison
Super Guru

Created ‎11-19-2019 11:02 AM

Not sure, it is your folder/files.

 

If you are not sure yourself,  just make another folder inside of that folder and try to use it.   

 

When you setup the data node to an empty folder it should resolve your issue.

4,420 Views
0 Kudos

avatar
author-rank Cico
Explorer

Created ‎11-20-2019 02:08 AM

I moved the content on another directory, restarted the namenode and the error message is gone.

So I could delete the old directory

4,400 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements