Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to retrive file from datanode

Highlighted

unable to retrive file from datanode

Hello all,

I started name node and data node in my cluster. I can read the list of files available in my hdfs file space. But unable to retrieve the file content. I got "BlockMissingException" while performing this action. Can any one help me to fix this issue.

org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1687189467-127.0.0.1-1459176139681:blk_1073742455_1634 file=/user/output/part-00000 at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896) at java.io.DataInputStream.read(DataInputStream.java:100) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:107) at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:102) at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) cat: Could not obtain block: BP-1687189467-127.0.0.1-1459176139681:blk_1073742455_1634 file=/user/output/part-00000

Thanks in advance.

Radhakrishnan RK.

3 REPLIES 3
Highlighted

Re: unable to retrive file from datanode

Mentor

What is the output of

hdfs fsck
Highlighted

Re: unable to retrive file from datanode

Rising Star
@Radhakrishnan Rk

From the error stack, following can be observed:

1. The File under issue is - /user/output/part-00000

2. This file is made of block - "blk_1073742455_1634" and Block "blk_1073742455_1634" is missing.

3. The "Block "blk_1073742455_1634" part of the BlockPool - "BP-1687189467-127.0.0.1-1459176139681", which is on your localhost.

To verify if the block is missing from the HDFS filesystem, you need to collect the output of:

# hdfs fsck / -files -blocks -locations

Highlighted

Re: unable to retrive file from datanode

New Contributor

I faced the same issue while trying to read hdfs file remotely from my IntelliJ IDE. You will be able to get handle of that hdfs file since its with hadoop namenode and you are connected to edge node (namenode) via kinit command. However, when you try to read hdfs file it tries to connect directly to datanode where your hdfs block is present (using the information given by namenode). There is no connectivity from your local machine (IDE) to the datanode, hence it wont be able to read the hdfs file


If your cluster is small enough try adding following properties in your hdfs-site.xml :


dfs.datanodes.address,dfs.datanodes.ipc.address,dfs.datanodes.http.address,dfs.datanodes.https.address


it should solve your problem.

Don't have an account?
Coming from Hortonworks? Activate your account here