well my bad, it turned out to be a connection issue as per the following log from executor. WARN DFSClient: Failed to connect to sandbox.hortonworks.com/<<IP>>:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused Here is what I did. on HDP 2.5 , using root login I modified start_scripts/start_sandbox.sh and forwarded the port 50010. and run the following commands docker commit sandbox sandbox<br> docker stop sandbox docker rm sandbox init 6 ( to restart ) now the spark master from Dev (A) can get the block from Dev (B) which is my HDP 2.5 machine. ... View more