Reply
Explorer
Posts: 6
Registered: ‎09-03-2015

Distcp error among two secured clusters.-socket exception

[ Edited ]

Hi All,

 

I have configured cross realm among 2 secured clusters.

 

hadoop fs -ls hdfs://remotenn:8020/  ----->able to see files.

hadoop fs -cp hdfs://source/path hdfs://dest/path worked from client only as hadoop.namenode.kerberos.pattern -* enabled only here.

 

If I try to run above 2 commands they run only if namenode.kerberos.pattern is enabled in hdfs-site.xml.

 

a)Shall I set this parameter through out the cluster?

b)Do i need to set dfs.socket.timeout and dfs.datanode.socket.write.timeout to some value.Currently write is set to 0 and read (socket.timeout) not set.

 

 

ERROR WHILE DISTCP COMMAND with current set up.

 

when I do hadoop distcp --->

 

It fails with below error.I have checked dfs.socket.timeout and dfs.datanode.socket.write.timeout in hdfs-site.xml ->both set to 0

 

15/08/31 19:02:57 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW

15/08/31 19:02:59 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1440506170094_2185 is still in NEW

15/08/31 19:03:00 INFO impl.YarnClientImpl: Submitted application application_1440506170094_2185

15/08/31 19:03:00 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/bdauser/.staging/job_1440506170094_2185

15/08/31 19:03:00 ERROR tools.DistCp: Exception encountered

java.io.IOException: Failed to run job : Call From gvlhdmpap02/39.7.56.2 to 10.234.22.56:8020 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending local=/39.7.56.2:42716 remote=10.234.22.56/10.234.22.56:8020]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout

        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)

        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)

        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)

        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)

        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)

        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)

        at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

        at org.apache.hadoop.tools.DistCp.main(DistCp.java:375)

bdauser@gvlengpap01:~>

 

 

IT guys say no firewall there among prod and nonprod.

 

Please suggest.

 

Regards,

Rajat

 

Explorer
Posts: 15
Registered: ‎11-27-2017

Re: Distcp error among two secured clusters.-socket exception

Hi Rajat,

 

Could you please help me upto setup Cross Realm trust?

 

 

Thanks,

Udai

Highlighted
Explorer
Posts: 23
Registered: ‎03-07-2017

Re: Distcp error among two secured clusters.-socket exception

Are your clusters TLS enabled? Do you see any SSL related exceptions?

Announcements