CANT copy files from my LAPTOP to HDFS

[#1]

Hi, I need to copy files to an HDFS from my laptop but I am getting this error. If I copy files inside the hadoop server all goes well. But outside the cluster/server I get this error. IT IS DRIVING ME CRAZY. I installed 3 different clusters and same error. I changed configuration but I cannot make it work.




./hadoop fs -copyFromLocal testasd.txt hdfs://server1:9000/testing.txt
2019-05-02 18:26:07,445 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-05-02 18:27:09,547 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073741864_1040
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.17.0.2:50010]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
2019-05-02 18:27:09,555 WARN hdfs.DataStreamer: Abandoning BP-581371184-172.17.13.14-1437578119536:blk_1073741864_1040
2019-05-02 18:27:09,885 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[172.17.0.2:50010,DS-a38d90cd-97c9-499b-914f-c710b2f0574a,DISK]
2019-05-02 18:27:10,058 WARN hdfs.DataStreamer: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /testing.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3067)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:722)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
at org.apache.hadoop.ipc.Client.call(Client.java:1457)
at org.apache.hadoop.ipc.Client.call(Client.java:1367)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:510)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.addBlock(DFSOutputStream.java:1081)
at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1866)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1668)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
copyFromLocal: File /testing.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.

[#2]

@Mariano Gastaldi

That seems strange but maybe explicable just because you have no hdfs client software on your laptop if you want to copy from your laptop unless its part of the cluster e.g an edge node then I don't see how you will succeed.

What's happening is you don't have the Hadoop client libraries installed usually if you want to interact with the cluster you install the client software like hdfs,YARN,sqoop,hbase clients etc.

Despite having an ubuntu or Linux based client there is no way you can copy those files unless you copy using something like WinSCP to your /home/analyst1 or /tmp in your home directory on a node which is part of the cluster !!!! then and ONLY then can you run the hdfs command from your local directory and not encounter "Unable to load native-hadoop library for your platform "

$ hdfs dfs -copyFromLocal file_to_copy.txt hdfs://server1:9000/testing.txt

For the below the directory /user/analyst1 should be pre-created with the correct permissions for user analyst1

$ hdfs dfs -copyFromLocal  sfile_to_copy.txt  /user/analyst1

Another solution is to deploy HDP standalone cluster then remove all the other components and copy the hdfs-site.xml and core-site.xml to your local laptop and initiate the connection.

Hope that helps

[#3]

I was able to fix it adding this in my hdfs-site files

Also I have added the permission line to avoid issues in the dev environment.

--------------HDFS-SITE.XML (SERVER)-----------
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
------------------------------------------------

I added the config below in my laptop (hadoop client)
----------------hdfs-site (CLIENT)---------------

<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>
--------------------------------------------------

[#4]

@Mariano Gastaldi

Good it worked !! so you installed the hdfs client just copied the hdfs-site.xml.

If you found this answer addressed your question, please take a moment to log in and click the "accept" link on the answer.

That would be a great help to Community users to find the solution quickly for these kinds of errors.