Cannot copy from local machine to VM datanode via Java

[#1]

Hello,

I have an application that copies data to HDFS, but is failing due to the datanode being excluded. See snippet:

private void copyFileToHdfs(FileSystem hdfs, Path localFilePath, Path hdfsFilePath) throws IOException, InterruptedException {
    log.info("Copying " + localFilePath + " to " + hdfsFilePath);
    hdfs.copyFromLocalFile(localFilePath, hdfsFilePath);
}

However, when I try to execute, I get:

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/dev/workflows/test.jar could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1583)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3109)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3033)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)

HDFS commands work fine. I can create, modify, delete. Copying data is the only problem. I've also attempted to check the ports by telnet'ing. I can telnet 8020, but not 50010. I assume this is the root of the issue and why the single datanode is being excluded. I attempted to add an iptable firewall rule but I still am running into the same issue.

Any help is appreciated.

[#2]

@Jim Fratzke

See this Link

[#3]

No effect on the problem.

[#4]

@Jim Fratzke

Please follow these suggestions. Link

You have to make sure that DN is running and there is connectivity between NN and DN.

[#5]

My dfsadmin -report shows space availability and I if I SSH onto the VM and copy data from the VM to HDFS, it works fine. It's only when I attempt to copy from my host machine to the VM hdfs that causes the problem.

[#6]

@Jim Fratzke

Option 1

a) Add your hostname to conf/slaves and retry!

Option 2

There are cases when a Data Node may not available to Name Node and the causes could be :

a)Data Node is Busy with block scanning and reporting

b)Data Node disk is Full

c)Check that the dfs.block.size value in hdfs-site.xml is not negative

d) check the diskspace in your system and make sure the logs are not warning you about it

e)while write in progress primary datanode goes down(Any n/w fluctations b/w Name Node and Data Node Machines)

when Ever we append any partial chunk and call sync for subsequent partial chunk appends client should store the previous data in buffer.

Option3 Last resort

The below procedure will destroy ALL data on HDFS. Do not execute the steps in this answer unless you do not care about destroying existing data!!

a)stop all hadoop services

b)delete dfs/name and dfs/data directories

c)hadoop namenode -format # Answer with a capital Y

d)start hadoop services

e)cluster's" dfs health page http://your_host:50070/dfshealth.jsp

[#7]

@Jim Fratzke what does your datanode log say?

[#8]

Not sure if it's related, but this shows up:

2016-02-14 18:15:19,232 ERROR datanode.DataNode (DataXceiver.java:run(278)) - sandbox.hortonworks.com:50010:DataXceiver error processing unknown operation  src: /127.0.0.1:55605 dst: /127.0.0.1:50010
java.io.EOFException
	at java.io.DataInputStream.readShort(DataInputStream.java:315)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
	at java.lang.Thread.run(Thread.java:745)

[#9]

@Jim Fratzke

The -copyFromLocal copies file from sandbox to hdfs not from windows to sandbox.

You need to scp the file from windows to sandbox using the scp command in the sandbox. Use Winscp for copying the files from windows to any unix based server like Hortonwork Sandbox.

[#10]

I can scp files onto the VM and use "hadoop fs -copyFromLocal" to move the file onto HDFS. That works fine. It's when I run the java code on my local machine and attempt to copy data using the filesystem object passed to me. I can run commands like hdfs.mkdirs() and hdfs.delete(), but hdfs.copyFromLocalFile() failed due to the described above.