I have a 3 node cluster (ml1, ml2, and vortex). ml1 is the namenode, ml2 and vortex are datanodes. Last week we had a power shutdown in our school and since then i have not been able to access the files in hdfs.
When i try to see the "Head of the file" i get an error: "Couldn't find datanode to read file from. Forbidden".
The size of the file is correct (my understanding is that the file is still somewhere). The output of command: hdfs dfsadmin -report is as below:
Configured Capacity: 2088742875136 (1.90 TB)
Present Capacity: 1673267032064 (1.52 TB)
DFS Remaining: 942202814464 (877.49 GB)
DFS Used: 731064217600 (680.86 GB)
DFS Used%: 43.69%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 6303
Missing blocks (with replication factor 1): 6303
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 10.3.0.38:9866 (vortex.serv)
Hostname: vortex.dbwks.erau.edu
Decommission Status : Normal
Configured Capacity: 1031987826688 (961.11 GB)
DFS Used: 531036213248 (494.57 GB)
Non DFS Used: 5448654848 (5.07 GB)
DFS Remaining: 443057381376 (412.63 GB)
DFS Used%: 51.46%
DFS Remaining%: 42.93%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Thu May 29 13:42:03 EDT 2025
Last Block Report: Never
Num of Blocks: 0
Name: 10.3.0.58:9866 (wxml2.serv)
Hostname: wxml2.db.erau.edu
Decommission Status : Normal
Configured Capacity: 1056755048448 (984.18 GB)
DFS Used: 200028004352 (186.29 GB)
Non DFS Used: 303877742592 (283.01 GB)
DFS Remaining: 499145433088 (464.87 GB)
DFS Used%: 18.93%
DFS Remaining%: 47.23%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Thu May 29 13:42:04 EDT 2025
Last Block Report: Thu May 29 12:56:07 EDT 2025
Num of Blocks: 3519
when i run command: hdfs fsck /hdfs_nifi/tfms/flightData/flModify/2023_03.csv -files -blocks -locations i get the error bwlow:
FileSystem is inaccessible due to:
java.io.FileNotFoundException: File does not exist: /hdfs_nifi/tfms/flightData/flModify/2023_03.csv
DFSck exiting.
But the file is visible under the above path. if i run command : hdfs dfs -ls /hdfs_nifi/tfms/flowInformation/Reroute/rrRouteData/ i get following:
Found 42 items
-rw-r--r-- 1 root supergroup 60193632 2025-04-26 21:01 /hdfs_nifi/tfms/flowInformation/Reroute/rrRouteData/2022_01.csv
-rw-r--r-- 1 root supergroup 120930719 2025-03-29 07:16 /hdfs_nifi/tfms/flowInformation/Reroute/rrRouteData/2022_02.csv
-rw-r--r-- 1 root supergroup 121990303 2025-04-30 16:37 /hdfs_nifi/tfms/flowInformation/Reroute/rrRouteData/2022_03.csv
-rw-r--r-- 1 root supergroup 116003694 2025-04-01 18:15 /hdfs_nifi/tfms/flowInformation/Reroute/rrRouteData/2022_04.csv
-rw-r--r-- 1 root supergroup 123227494 2025-05-04 08:24 /hdfs_nifi/tfms/flowInformation/Reroute/rrRouteData/2022_05.csv
I ran command: hdfs fsck -list-corruptfileblocks , then selected any random block from the output, and run command: find /data/hadoop_dir/datanode_dir/ -type f -iname blk_1073786680 , and then go to the location returned (/data/hadoop_dir/datanode_dir/current/BP-1329719554-155.31.114.228-1716835341630/current/finalized/subdir0/subdir15/blk_1073786680) i do see data block there but why am i not able to access the data in HDFS?
Also, on one of the datanodes (vortex) i get following warning:
2025-05-29 17:43:47,616 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Unsuccessfully sent block report 0x40e0062fac3f698a with lease ID 0x8d10044dcd041620 to namenode: wxml1.serv/10.3.0.48:8020, containing 1 storage report(s), of which we sent 0. The reports had 6312 total blocks and used 0 RPC(s). This took 1 msecs to generate and 2 msecs for RPC and NN processing. Got back no commands.
2025-05-29 17:43:47,616 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.io.IOException): java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.runBlockOp(BlockManager.java:5558)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1651)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:182)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:34769)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1246)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1169)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3203)
Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at org.apache.hadoop.thirdparty.protobuf.IterableByteBufferInputStream.read(IterableByteBufferInputStream.java:143)
at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.read(CodedInputStream.java:2080)
i checked command: find $HADOOP_HOME -name "protobuf*.jar"
it returns following on all three machines:
/apache/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar
I would really appreciate if someone would suggest me something about this issue i.e., is there a way to get my data? I really need help with this.
