Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎07-23-2014
Accepted Solution

HDFS put failing due to internal IP address use

Hello,

 

I have an 7 node cluster (6 datanodes) and I am executing HDFSClient Put from an application outside the cloudera cluster. Internally the

cluster is configured to use an internal IP (172.x.x.x range)

 

I get the error below when I issue a hdfs put to the name node ... note the IP returned is172.123.123.123:50010 which is the internal ip address

and not accessible from the application host. 

 HDFS

2015-09-22 19:26:48.292+01:00 INFO [Thread-11] org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.123.123.123:50010]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534) ~[hadoop-common-2.7.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1610) ~[hadoop-hdfs-2.6.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) [hadoop-hdfs-2.6.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) [hadoop-hdfs-2.6.0.jar!/:na]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) [hadoop-hdfs-2.6.0.jar!/:na]
2015-09-22 19:26:48.292+01:00 INFO [Thread-11] org.apache.hadoop.hdfs.DFSClient - Abandoning BP-383231650-172.16.1.45-1435792324508:blk_1074408707_667883
2015-09-22 19:26:48.316+01:00 INFO [Thread-11] org.apache.hadoop.hdfs.DFSClient - Excluding datanode 172.123.123.123:50010

 

Wildcard addresses is being used on datanode/namenode

Also, I've tried enabling the following parameter to no avail:

 

dfs.datanode.use.datanode.hostname
dfs.client.use.datanode.hostname

 

 

 Is there any way to make the hostname be returned here instead of that IP?

 

Any pointers appreciated.. 

Brian

Posts: 1,537
Kudos: 277
Solutions: 234
Registered: ‎07-31-2013

Re: HDFS put failing due to internal IP address use

Wildcard addresses is being used on datanode/namenode

> dfs.client.use.datanode.hostname

 

This is your solution here, iff your client hosts will resolve the very same DN hostname but over a different IP. Is that true in your environment?

 

You mention you've tried this - could you elaborate? This setting needs to be applied at the HDFS client configuration, for it to be properly in effect. Is your 'edge host' that lies out of the cluster, or your Java application (if it is run standalone), configured with this set to true in its hdfs-site.xm/Configuration object?

Backline Customer Operations Engineer
New Contributor
Posts: 3
Registered: ‎07-23-2014

Re: HDFS put failing due to internal IP address use

FYI: I resolved this by setting the property on the client code....
Posts: 1,537
Kudos: 277
Solutions: 234
Registered: ‎07-31-2013

Re: HDFS put failing due to internal IP address use

Glad to hear! Please consider marking this thread as resolved so others with similar problems may find a solution quicker.
Backline Customer Operations Engineer
New Contributor
Posts: 3
Registered: ‎09-08-2016

Re: HDFS put failing due to internal IP address use

can you please elborrate ?  what u did

New Contributor
Posts: 2
Registered: ‎06-09-2017

Re: HDFS put failing due to internal IP address use

How. Which property in the client code?
Contributor
Posts: 31
Registered: ‎07-11-2017

Re: HDFS put failing due to internal IP address use

I have the same problem.

 

Could you please elaborate on the solution to solve this?

Announcements