question Cannot copy from local machine to VM datanode via Java in Support Questions

Cannot copy from local machine to VM datanode via Java

jfratzke — Sun, 14 Feb 2016 11:38:07 GMT

Hello,

I have an application that copies data to HDFS, but is failing due to the datanode being excluded. See snippet:

private void copyFileToHdfs(FileSystem hdfs, Path localFilePath, Path hdfsFilePath) throws IOException, InterruptedException {
    log.info("Copying " + localFilePath + " to " + hdfsFilePath);
    hdfs.copyFromLocalFile(localFilePath, hdfsFilePath);
}

However, when I try to execute, I get:

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/dev/workflows/test.jar could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1583)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3109)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3033)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)

HDFS commands work fine. I can create, modify, delete. Copying data is the only problem. I've also attempted to check the ports by telnet'ing. I can telnet 8020, but not 50010. I assume this is the root of the issue and why the single datanode is being excluded. I attempted to add an iptable firewall rule but I still am running into the same issue.

Any help is appreciated.

Re: Cannot copy from local machine to VM datanode via Java

nsabharwal — Sun, 14 Feb 2016 11:47:11 GMT

@Jim Fratzke

See this Link

Re: Cannot copy from local machine to VM datanode via Java

jfratzke — Sun, 14 Feb 2016 14:51:26 GMT

No effect on the problem.

Re: Cannot copy from local machine to VM datanode via Java

Shelton — Sun, 14 Feb 2016 16:01:32 GMT

@Jim Fratzke

Option 1

a) Add your hostname to conf/slaves and retry!

Option 2

There are cases when a Data Node may not available to Name Node and the causes could be :

a)Data Node is Busy with block scanning and reporting

b)Data Node disk is Full

c)Check that the dfs.block.size value in hdfs-site.xml is not negative

d) check the diskspace in your system and make sure the logs are not warning you about it

e)while write in progress primary datanode goes down(Any n/w fluctations b/w Name Node and Data Node Machines)

when Ever we append any partial chunk and call sync for subsequent partial chunk appends client should store the previous data in buffer.

Option3 Last resort

The below procedure will destroy ALL data on HDFS. Do not execute the steps in this answer unless you do not care about destroying existing data!!

a)stop all hadoop services

b)delete dfs/name and dfs/data directories

c)hadoop namenode -format # Answer with a capital Y

d)start hadoop services

e)cluster's" dfs health page http://your_host:50070/dfshealth.jsp

Re: Cannot copy from local machine to VM datanode via Java

aervits — Sun, 14 Feb 2016 20:40:23 GMT

@Jim Fratzke what does your datanode log say?

Re: Cannot copy from local machine to VM datanode via Java

nsabharwal — Sun, 14 Feb 2016 20:51:33 GMT

@Jim Fratzke

Please follow these suggestions. Link

You have to make sure that DN is running and there is connectivity between NN and DN.

Re: Cannot copy from local machine to VM datanode via Java

jfratzke — Mon, 15 Feb 2016 02:16:05 GMT

Not sure if it's related, but this shows up:

2016-02-14 18:15:19,232 ERROR datanode.DataNode (DataXceiver.java:run(278)) - sandbox.hortonworks.com:50010:DataXceiver error processing unknown operation  src: /127.0.0.1:55605 dst: /127.0.0.1:50010
java.io.EOFException
	at java.io.DataInputStream.readShort(DataInputStream.java:315)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
	at java.lang.Thread.run(Thread.java:745)

Re: Cannot copy from local machine to VM datanode via Java

jfratzke — Mon, 15 Feb 2016 02:18:19 GMT

My dfsadmin -report shows space availability and I if I SSH onto the VM and copy data from the VM to HDFS, it works fine. It's only when I attempt to copy from my host machine to the VM hdfs that causes the problem.

Re: Cannot copy from local machine to VM datanode via Java

Shelton — Mon, 15 Feb 2016 02:43:41 GMT

@Jim Fratzke

The -copyFromLocal copies file from sandbox to hdfs not from windows to sandbox.

You need to scp the file from windows to sandbox using the scp command in the sandbox. Use Winscp for copying the files from windows to any unix based server like Hortonwork Sandbox.

Re: Cannot copy from local machine to VM datanode via Java

nsabharwal — Mon, 15 Feb 2016 04:33:26 GMT

@Jim Fratzke

Take a look on this tutorial.. http://stackoverflow.com/questions/12783968/copying-directory-from-local-system-to-hdfs-java-code

Re: Cannot copy from local machine to VM datanode via Java

jfratzke — Mon, 15 Feb 2016 05:10:15 GMT

I can scp files onto the VM and use "hadoop fs -copyFromLocal" to move the file onto HDFS. That works fine. It's when I run the java code on my local machine and attempt to copy data using the filesystem object passed to me. I can run commands like hdfs.mkdirs() and hdfs.delete(), but hdfs.copyFromLocalFile() failed due to the described above.

Re: Cannot copy from local machine to VM datanode via Java

aervits — Mon, 15 Feb 2016 08:15:11 GMT

@Jim Fratzke

check firewall, in vm settings make sure allow all is enabled for network adapter, your vm fqdn is in /etc/hosts, your hdfs scheme in your URL in your code is properly identified and not recognizing local filesystem. Also hdfs Port on Sandbox is 8020 not 50010.

Re: Cannot copy from local machine to VM datanode via Java

jfratzke — Mon, 15 Feb 2016 11:30:25 GMT

VM settings was part of it. As soon as I added port 50010, I got some breakthrough. There's now some other issue where the dataQueue.wait() gets stuck in DFSOutputStream. Also the namenode listens on 8020 which is working fine as stated above. The problem was connecting to the datanodes which I can connect to now.

Re: Cannot copy from local machine to VM datanode via Java

aervits — Mon, 15 Feb 2016 11:50:58 GMT

@Jim Fratzke that's good, progress, please post logs. According to src, its waiting for acks so it is still may look like networking issue.

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-hdfs/0.23.1/org/apache/hadoop/hdfs/DFSOutputStream.java

Re: Cannot copy from local machine to VM datanode via Java

jfratzke — Mon, 15 Feb 2016 13:22:58 GMT

Ok, a bit more progress. Digging into DFSOutputStream.createSocketForPipeline, the ip address for the datanode seems to be resolving to the internal ip 10.0.2.15, however I do have dfs.client.use.datanode.hostname set to true so I'm not sure why it's attempting to connect to that. My host machine has the hostname sandbox.hortonworks.com pointed at 127.0.0.1, however on the VM that hostname is pointed at 10.0.2.15.

Re: Cannot copy from local machine to VM datanode via Java

aervits — Tue, 16 Feb 2016 03:31:47 GMT

@Jim Fratzke

I'd be curious if you ran dhclient command once you ssh as root. Do you have a 2nd adapter added to the VM? If not, add it in VM network settings and make sure allow all is set for that adapter as well, then once it boots up, run dhclient command.

Re: Cannot copy from local machine to VM datanode via Java

jfratzke — Wed, 17 Feb 2016 00:56:48 GMT

So, the problem was two issues. One was that the VM does not have port 50010 opened by default, so all the datanodes are excluded leading to the issue above. The other issue was I needed to set "dfs.client.use.datanode.hostname" to true to avoid the datanodes resolving to the internal ip on the VM which I did set. Finally, after stepping through the configuration, I found that it was still being set to "false" which turned out to be a problem in my own code. My FileSystem object was being created with a new Configuration() rather than the one I had loaded with the configs from hdfs-site and core-site pulled from Ambari. Whoops!

Anyway, Thanks for the help all.

Re: Cannot copy from local machine to VM datanode via Java

jazzfan159 — Sat, 20 Aug 2016 03:06:32 GMT

I had to do more task on OSX El Capitan: in /etc/hosts, add this entry:

127.0.0.1    sandbox.hortonworks.com

Without it, you will see a java.nio.channels.UnresolvedAddressException

Re: Cannot copy from local machine to VM datanode via Java

mustafa_hasanbu — Mon, 20 Nov 2017 08:56:50 GMT

I understand this is a while ago, but can you remember how you opened the port 50010 on the VM? Hortonworks has a tutorial on opening ports in HDP Sandbox box, but it is actually not working. After stopping the docker container, I cannot actually remove the container because connection is being refused.