org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block even if the file is present.

[#1]

I am trying to run the spark job with hive support enabled. it can run the command "show databases" successfully but when it try to read hive table ( which had data stored as txt on hdfs ) is showing org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:....

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 3.0 (TID 6, 192.168.8.134, executor 0): org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: <bloac ID >> =<<path>>
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:984)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)

Here are the details of my dev environment:

Dev box (A) (centOS running in vmware ) with eclipse added jars from spark 2.2.1 with hadoop 2.7 support.

Dev box (A) is running with spark master and slave configured with thrift server on Dev box (B) . Dev box (B) is running HDP 2.5 hortonworks.

So, why the app running in dev box (A) is throwing the missing block exception when it try to query hive table , even if the file is present in hdfs ?

Please note I have already executed the following command to check for blocks.

sudo -u hdfs hdfs dfsadmin -report

sudo -u hdfs hdfs fsck -list-corruptfileblocks

Thanks for any help!

[#2]

well my bad, it turned out to be a connection issue as per the following log from executor.

WARN DFSClient: Failed to connect to sandbox.hortonworks.com/<<IP>>:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused

Here is what I did.

on HDP 2.5 , using root login I modified start_scripts/start_sandbox.sh and forwarded the port 50010.

and run the following commands

docker commit sandbox sandbox<br>

docker stop sandbox
docker rm sandbox
init 6 ( to restart ) 

now the spark master from Dev (A) can get the block from Dev (B) which is my HDP 2.5 machine.