Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Distcp among two secured HA clusters failing with socket exception

Highlighted

Distcp among two secured HA clusters failing with socket exception

Explorer

Hi All,

 

Thanks Harsh for the support so far.


I have configured cross realm among 2 secured clusters.

hadoop fs -ls hdfs://remotenn:8020/ ----->able to see files.
hadoop fs -cp hdfs://source/path hdfs://dest/path worked from client only as hadoop.namenode.kerberos.pattern -* enabled only here.

If I try to run above 2 commands they run only if namenode.kerberos.pattern is enabled in hdfs-site.xml.

a)Shall I set this parameter through out the cluster?
b)Do i need to set dfs.socket.timeout and dfs.datanode.socket.write.timeout to some value.Currently write is set to 0 and read (socket.timeout) not set.


ERROR WHILE DISTCP COMMAND with current set up.

when I do hadoop distcp --->

It fails with below error.I have checked dfs.socket.timeout and dfs.datanode.socket.write.timeout in hdfs-site.xml ->both set to 0

15/08/31 19:02:57 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW
15/08/31 19:02:59 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW
15/08/31 19:03:00 INFO impl.YarnClientImpl: Submitted application application_1440506170094_2185
15/08/31 19:03:00 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/bdauser/.staging/job_1440506170094_2185
15/08/31 19:03:00 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to run job : Call From gvlhdmpap02/39.7.56.2 to 10.234.22.56:8020 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending local=/39.7.56.2:42716 remote=10.234.22.56/10.234.22.56:8020]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:375)
bdauser@gvlengpap01:~>


IT guys say no firewall there among prod and nonprod.

Please suggest.

Regards,
Rajat

2 REPLIES 2

Re: Distcp among two secured HA clusters failing with socket exception

Explorer
Suggestion Admins ..!!!

Re: Distcp among two secured HA clusters failing with socket exception

Explorer

map reduce not working to remote realm.

as job not getting submitted failing with same distcp error..

Do I need to check hadoop.security_auth.to.local parameter ? as hadoop fs -cp hdfs://localNN hdfs://remoteNN/ is working good.


@Rajat wrote:
Suggestion Admins ..!!!

 

Don't have an account?
Coming from Hortonworks? Activate your account here