Hadoop datanode is down after power outage

[#1]

Datanode is down after power outage and few files were corrupted

I have deleted the corrupted files and tried to start the datanode "hdfs fsck / -delete"

Datanode logs were something like this:

2019-06-12 14:18:55,659 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Caught exception while scanning /home/hadoop/hadoopdata/hdfs/datanode/current. Will throw later. ExitCodeException exitCode=1: du: cannot access ‘/home/hadoop/hadoopdata/hdfs/datanode/current/BP-354460020-192.168.42.27-1473368397287/current/finalized/subdir122/subdir192/blk_1098532097_24912327.meta’: Structure needs cleaning du: cannot access ‘/home/hadoop/hadoopdata/hdfs/datanode/current/BP-354460020-192.168.42.27-1473368397287/current/finalized/subdir122/subdir192/blk_1098532097’: Structure needs cleaning du: cannot access ‘/home/hadoop/hadoopdata/hdfs/datanode/current/BP-354460020-192.168.42.27-1473368397287/current/finalized/subdir122/subdir192/blk_1098532113_24912343.meta’: Structure needs cleaning 2019-06-12 14:18:55,671 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool (Datanode Uuid unassigned) service to hadoop1/192.168.42.27:9000 2019-06-12 14:18:55,672 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool (Datanode Uuid unassigned) 2019-06-12 14:18:57,673 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode 2019-06-12 14:18:57,675 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0 2019-06-12 14:18:57,693 INFO org.apache.hadoop.hdfs.server.datanode.

/************************************************************

SHUTDOWN_MSG: Shutting down DataNode at hadoop18/192.168.42.57

************************************************************/

can someone please help me with the solution

[#2]

@Nikhila M

The error is being thrown is because it can't access the /home/hadoop/hadoopdata/* After the reboot of the cluster it mounted a /home which is not valid according to hadoop as a data dir.

This is as per Ambari design that you can not use the "/home" directory for NameNode/ DataNode data directory. Have a lot at this jira https://issues.apache.org/jira/browse/AMBARI-4162 can you check the values of dfs.datanode.data.dir

Solution is to remove or change /home/hadoop/hadoopdata/hdfs/datanode/current to /hadoop/hadoopdata/hdfs/datanode/current

You should be able to restart the datanode after making the above change.

Please revert

[#3]

@Nikhila M

Any updates, please if the issue is resolved take some time and "accept" the answer and share your solution so that members who encounter the same problem can learn from your experience like giving back to the community :-)

Please revert