The DataNode has 1 volume failure but disks seem fine

Cico — Wed, 20 Nov 2019 00:33:32 GMT

I added an old server already used as datanode in my cluster.

The process get completed, but as soon as the datanode is been added I get the following error message on cloudera manager:

"The DataNode has 1 volume failure but disks"

I don't have any error message, just one warning as soon as I added the host:

5:27:37.578 PM WARN Storage Failed to add storage directory [DISK]file:/data/2/dfs/dn/ org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/2/dfs/dn is in an inconsistent state: Can't format the storage directory because the current/ directory is not empty. at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:495) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:600) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:279) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:418) at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1560) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1520) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:354) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) at java.lang.Thread.run(Thread.java:748)

and this are the common INFO log from that host:

5:40:11.726 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685 for deletion 5:40:11.727 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638685_4897872 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638685 5:42:14.918 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874 src: /10.179.104.167:55132 dest: /10.179.104.168:50010 5:42:14.937 PM INFO clienttrace src: /10.179.104.167:55132, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1219924065_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, duration: 12451111 5:42:14.937 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638687_4897874, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 5:42:17.729 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687 for deletion 5:42:17.731 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638687_4897874 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638687 5:43:14.919 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875 src: /10.179.104.167:55140 dest: /10.179.104.168:50010 5:43:14.938 PM INFO clienttrace src: /10.179.104.167:55140, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-879119697_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, duration: 12054137 5:43:14.938 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638688_4897875, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 5:43:17.732 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688 for deletion 5:43:17.733 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638688_4897875 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638688 5:44:14.936 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876 src: /10.179.104.165:56144 dest: /10.179.104.168:50010 5:44:14.950 PM INFO clienttrace src: /10.179.104.165:56144, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_573364305_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, duration: 11591201 5:44:14.951 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638689_4897876, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:44:17.736 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689 for deletion 5:44:17.738 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638689_4897876 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638689 5:46:14.937 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878 src: /10.179.104.165:56158 dest: /10.179.104.168:50010 5:46:14.951 PM INFO clienttrace src: /10.179.104.165:56158, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_562798393_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, duration: 11582892 5:46:14.951 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638691_4897878, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:46:17.744 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691 for deletion 5:46:17.746 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638691_4897878 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638691 5:48:14.941 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880 src: /10.179.104.167:55180 dest: /10.179.104.168:50010 5:48:14.957 PM INFO clienttrace src: /10.179.104.167:55180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1107939178_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, duration: 13081545 5:48:14.958 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638693_4897880, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:48:20.748 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693 for deletion 5:48:20.750 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638693_4897880 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638693 5:49:19.944 PM INFO DataNode Receiving BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881 src: /10.179.104.165:56180 dest: /10.179.104.168:50010 5:49:19.959 PM INFO clienttrace src: /10.179.104.165:56180, dest: /10.179.104.168:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1732671963_103, offset: 0, srvID: 236da03c-1676-4097-a3a7-17a3f308008c, blockid: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, duration: 12530220 5:49:19.960 PM INFO DataNode PacketResponder: BP-22824834-10.179.104.198-1543004359329:blk_1078638694_4897881, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 5:49:26.752 PM INFO FsDatasetAsyncDiskService Scheduling blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694 for deletion 5:49:26.754 PM INFO FsDatasetAsyncDiskService Deleted BP-22824834-10.179.104.198-1543004359329 blk_1078638694_4897881 file /data/1/dfs/dn/current/BP-22824834-10.179.104.198-1543004359329/current/finalized/subdir74/subdir184/blk_1078638694

Any help on going deeply with this error?

Re: The DataNode has 1 volume failure but disks seem fine

stevenmatison — Tue, 19 Nov 2019 18:14:33 GMT

First thing I see is:

Can't format the storage directory because the current/ directory is not empty.

Is the path/folder empty? look for hidden files too...

Re: The DataNode has 1 volume failure but disks seem fine

Cico — Tue, 19 Nov 2019 18:17:50 GMT

It's not empty.

Is it safe to delete the content?

Re: The DataNode has 1 volume failure but disks seem fine

stevenmatison — Tue, 19 Nov 2019 19:02:26 GMT

Not sure, it is your folder/files.

If you are not sure yourself, just make another folder inside of that folder and try to use it.

When you setup the data node to an empty folder it should resolve your issue.

Re: The DataNode has 1 volume failure but disks seem fine

Cico — Wed, 20 Nov 2019 10:08:07 GMT

I moved the content on another directory, restarted the namenode and the error message is gone.

So I could delete the old directory

question Re: The DataNode has 1 volume failure but disks seem fine in Support Questions

The DataNode has 1 volume failure but disks seem fine

Re: The DataNode has 1 volume failure but disks seem fine

Re: The DataNode has 1 volume failure but disks seem fine

Re: The DataNode has 1 volume failure but disks seem fine

Re: The DataNode has 1 volume failure but disks seem fine