question Re: HDFS put failing due to internal IP address use in Archives of Support Questions (Read Only)

HDFS put failing due to internal IP address use

briandoran — Tue, 22 Sep 2015 19:21:08 GMT

Hello,

I have an 7 node cluster (6 datanodes) and I am executing HDFSClient Put from an application outside the cloudera cluster. Internally the

cluster is configured to use an internal IP (172.x.x.x range)

I get the error below when I issue a hdfs put to the name node ... note the IP returned is172.123.123.123:50010 which is the internal ip address

and not accessible from the application host.

HDFS

2015-09-22 19:26:48.292+01:00 INFO [Thread-11] org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.123.123.123:50010]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534) ~[hadoop-common-2.7.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1610) ~[hadoop-hdfs-2.6.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) [hadoop-hdfs-2.6.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) [hadoop-hdfs-2.6.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) [hadoop-hdfs-2.6.0.jar!/:na]
2015-09-22 19:26:48.292+01:00 INFO [Thread-11] org.apache.hadoop.hdfs.DFSClient - Abandoning BP-383231650-172.16.1.45-1435792324508:blk_1074408707_667883
2015-09-22 19:26:48.316+01:00 INFO [Thread-11] org.apache.hadoop.hdfs.DFSClient - Excluding datanode 172.123.123.123:50010

Wildcard addresses is being used on datanode/namenode

Also, I've tried enabling the following parameter to no avail:

dfs.datanode.use.datanode.hostname
dfs.client.use.datanode.hostname

Is there any way to make the hostname be returned here instead of that IP?

Any pointers appreciated..

Brian

Re: HDFS put failing due to internal IP address use

Harsh J — Wed, 23 Sep 2015 14:15:42 GMT

> Wildcard addresses is being used on datanode/namenode

> dfs.client.use.datanode.hostname

This is your solution here, iff your client hosts will resolve the very same DN hostname but over a different IP. Is that true in your environment?

You mention you've tried this - could you elaborate? This setting needs to be applied at the HDFS client configuration, for it to be properly in effect. Is your 'edge host' that lies out of the cluster, or your Java application (if it is run standalone), configured with this set to true in its hdfs-site.xm/Configuration object?

Re: HDFS put failing due to internal IP address use

briandoran — Thu, 01 Oct 2015 21:39:44 GMT

FYI: I resolved this by setting the property on the client code....

Re: HDFS put failing due to internal IP address use

Harsh J — Fri, 02 Oct 2015 02:11:07 GMT

Glad to hear! Please consider marking this thread as resolved so others with similar problems may find a solution quicker.

Re: HDFS put failing due to internal IP address use

manoj10 — Mon, 19 Sep 2016 15:53:31 GMT

can you please elborrate ? what u did

Re: HDFS put failing due to internal IP address use

RAVINDRABAJPAI — Sun, 11 Jun 2017 09:59:56 GMT

How. Which property in the client code?

Re: HDFS put failing due to internal IP address use

renuka20 — Wed, 02 Aug 2017 20:53:31 GMT

I have the same problem.

Could you please elaborate on the solution to solve this?

Re: HDFS put failing due to internal IP address use

krishnaM — Thu, 09 Nov 2017 20:39:34 GMT

Hi I am facing the same issue . Can anyone shed some light on this please

Thanks in Advance

Re: HDFS put failing due to internal IP address use

SandeshMendon — Thu, 28 Dec 2017 09:56:36 GMT

Can you please explain the steps involved to resolve this issue?

Re: HDFS put failing due to internal IP address use

mystreet — Mon, 30 Jul 2018 19:24:51 GMT

How did u reslove this issue...

Re: HDFS put failing due to internal IP address use

ddekany — Sat, 11 Aug 2018 09:18:23 GMT

Since it wasn't really described how exactly did you resolve it... The point is that on the client side (it's important that it's not on the server side), set "dfs.datanode.use.datanode.hostname" in the org.apache.hadoop.conf.Configuration object to value "true". If the Configuration object isn't created by your code (like if Spark creates it, in my case), then it depends on what creates it... see its documentation. But some guesses:

Attempt 1: Set it inside $HADOOP_HOME/etc/hadoop/hdfs-site.xml. Hadoop command line tools use that, your Java application though... maybe not.

Attempt 2: Put $HADOOP_HOME/etc/hadoop/ into the Java classpath (or pack hdfs-site.xml into your project under /src/main/resources/, but that's kind of dirty...). This works with Spark.

Spark only: SparkSession.builder().config("spark.hadoop.dfs.client.use.datanode.hostname", "true").[...]

Of course, you may also need to add the domain name of the DataNode-s (as the NameNode knows it) into the /etc/hosts on the computer running your application.