Reply
Highlighted
Explorer
Posts: 6
Registered: ‎09-03-2015

Distcp among two secured HA clusters failing with socket exception

Hi All,

 

Thanks Harsh for the support so far.


I have configured cross realm among 2 secured clusters.

hadoop fs -ls hdfs://remotenn:8020/ ----->able to see files.
hadoop fs -cp hdfs://source/path hdfs://dest/path worked from client only as hadoop.namenode.kerberos.pattern -* enabled only here.

If I try to run above 2 commands they run only if namenode.kerberos.pattern is enabled in hdfs-site.xml.

a)Shall I set this parameter through out the cluster?
b)Do i need to set dfs.socket.timeout and dfs.datanode.socket.write.timeout to some value.Currently write is set to 0 and read (socket.timeout) not set.


ERROR WHILE DISTCP COMMAND with current set up.

when I do hadoop distcp --->

It fails with below error.I have checked dfs.socket.timeout and dfs.datanode.socket.write.timeout in hdfs-site.xml ->both set to 0

15/08/31 19:02:57 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW
15/08/31 19:02:59 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW
15/08/31 19:03:00 INFO impl.YarnClientImpl: Submitted application application_1440506170094_2185
15/08/31 19:03:00 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/bdauser/.staging/job_1440506170094_2185
15/08/31 19:03:00 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to run job : Call From gvlhdmpap02/39.7.56.2 to 10.234.22.56:8020 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending local=/39.7.56.2:42716 remote=10.234.22.56/10.234.22.56:8020]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:375)
bdauser@gvlengpap01:~>


IT guys say no firewall there among prod and nonprod.

Please suggest.

Regards,
Rajat

Explorer
Posts: 6
Registered: ‎09-03-2015

Re: Distcp among two secured HA clusters failing with socket exception

Suggestion Admins ..!!!
Explorer
Posts: 6
Registered: ‎09-03-2015

Re: Distcp among two secured HA clusters failing with socket exception

map reduce not working to remote realm.

as job not getting submitted failing with same distcp error..

Do I need to check hadoop.security_auth.to.local parameter ? as hadoop fs -cp hdfs://localNN hdfs://remoteNN/ is working good.


@Rajat wrote:
Suggestion Admins ..!!!

 

Announcements