Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[FALCON] distcp failed firewall

[FALCON] distcp failed firewall

New Contributor

hi all,

do you know what are rules needed to open between two clusters hadoop to run correctly distcp ?

FALCON cluster entity needs to use port 50070 to make a distcp :

$  hadoop distcp -Dmapred.job.queue.name=oozie-launcher hftp://clusterProd:50070/tmp/falcon/currentProd-20160627/2016-06-27-12/motd-12 hdfs://clusterBackup:8020/tmp/motd-112
16/07/15 12:01:23 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hftp://clusterProd:50070/tmp/falcon/currentProd-20160627/2016-06-27-12/motd-12], targetPath=hdfs://clusterBackup:8020/tmp/motd-112, targetPathExists=false, preserveRawXattrs=false}
16/07/15 12:01:24 INFO impl.TimelineClientImpl: Timeline service address: http://clusterBackup:8188/ws/v1/timeline/
16/07/15 12:01:25 WARN token.Token: Cannot find class for token kind HFTP delegation
16/07/15 12:01:25 INFO security.TokenCache: Got dt for hftp://clusterProd:50070; Kind: HFTP delegation, Service: 10.99.224.105:50070, Ident: 00 06 66 61 6c 63 6f 6e 04 79 61 72 6e 00 8a 01 55 ee 00 ea 2c 8a 01 56 12 0d 6e 2c 8d 07 ac 21 8e 01 fc
16/07/15 12:01:25 INFO impl.TimelineClientImpl: Timeline service address: http://clusterBackup:8188/ws/v1/timeline/
16/07/15 12:01:25 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 11019 for falcon on 10.98.138.85:8020
16/07/15 12:01:25 INFO security.TokenCache: Got dt for hdfs://clusterBackup:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.98.138.85:8020, Ident: (HDFS_DELEGATION_TOKEN token 11019 for falcon)
16/07/15 12:01:25 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
16/07/15 12:01:25 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 11020 for falcon on ha-hdfs:bigdata-next
16/07/15 12:01:25 INFO security.TokenCache: Got dt for hdfs://bigdata-next; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 11020 for falcon)
16/07/15 12:01:26 INFO mapreduce.JobSubmitter: number of splits:1
16/07/15 12:01:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1467207300026_0044
16/07/15 12:01:26 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 11020 for falcon)
Kind: HFTP delegation, Service: 10.99.224.105:50070, Ident: 00 06 66 61 6c 63 6f 6e 04 79 61 72 6e 00 8a 01 55 ee 00 ea 2c 8a 01 56 12 0d 6e 2c 8d 07 ac 21 8e 01 fc
16/07/15 12:01:26 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 10.98.138.85:8020, Ident: (HDFS_DELEGATION_TOKEN token 11019 for falcon)
16/07/15 12:01:26 INFO impl.YarnClientImpl: Submitted application application_1467207300026_0044
16/07/15 12:01:26 INFO mapreduce.Job: The url to track the job: http://master002.next.rec.mapreduce.m1.p.fti.net:8088/proxy/application_1467207300026_0044/
16/07/15 12:01:26 INFO tools.DistCp: DistCp job-id: job_1467207300026_0044
16/07/15 12:01:26 INFO mapreduce.Job: Running job: job_1467207300026_0044
16/07/15 12:01:35 INFO mapreduce.Job: Job job_1467207300026_0044 running in uber mode : false
16/07/15 12:01:35 INFO mapreduce.Job:  map 0% reduce 0%
16/07/15 12:01:51 INFO mapreduce.Job:  map 100% reduce 0%
16/07/15 12:04:51 INFO mapreduce.Job: Task Id : attempt_1467207300026_0044_m_000000_0, Status : FAILED
Error: java.io.IOException: File copy failed: hftp://clusterProd:50070/tmp/falcon/currentProd-20160627/2016-06-27-12/motd-12 --> hdfs://clusterBackup:8020/tmp/motd-112
        at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:285)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hftp://clusterProd:50070/tmp/falcon/currentProd-20160627/2016-06-27-12/motd-12 to hdfs://clusterBackup:8020/tmp/motd-112
        at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
        at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
        ... 10 more
Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.net.SocketTimeoutException: connect timed out
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:302)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:247)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
        at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
        ... 11 more
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
        at sun.net.www.http.HttpClient.New(HttpClient.java:308)
        at sun.net.www.http.HttpClient.New(HttpClient.java:326)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
        at sun.net.www.protocol.http.HttpURLConnection.followRedirect0(HttpURLConnection.java:2662)
        at sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2584)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1770)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:370)
        at org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:135)
        at org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:116)
        at org.apache.hadoop.hdfs.web.ByteRangeInputStream.<init>(ByteRangeInputStream.java:101)
        at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:383)
        at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:388)
        at org.apache.hadoop.hdfs.web.HftpFileSystem.open(HftpFileSystem.java:404)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:298)
        ... 16 more

16/07/15 12:04:52 INFO mapreduce.Job:  map 0% reduce 0%
16/07/15 12:05:04 INFO mapreduce.Job:  map 100% reduce 0%

2 REPLIES 2
Highlighted

Re: [FALCON] distcp failed firewall

Cloudera Employee

In order to distcp between two HDFS HA cluster (for example A and B), modify the following in the hdfs-site.xml for both clusters:

For example, nameservice for cluster A and B is HAA and HAB respectively.

- Add value to the nameservice for both clusters dfs.nameservices = HAA, HAB

- Add property dfs.internal.nameservices

In cluster A:

dfs.internal.nameservices = HAA

In cluster B:

dfs.internal.nameservices = HAB

- Add dfs.ha.namenodes.<nameservice>

In cluster A

dfs.ha.namenodes.HAB = nn1,nn2

In cluster B

dfs.ha.namenodes.HAA = nn1,nn2

- Add property dfs.namenode.rpc-address.<cluster>.<nn>

In cluster A

dfs.namenode.rpc-address.HAB.nn1 = <NN1_fqdn>:8020

dfs.namenode.rpc-address.HAB.nn2 = <NN2_fqdn>:8020

In cluster B

dfs.namenode.rpc-address.HAA.nn1 = <NN1_fqdn>:8020

dfs.namenode.rpc-address.HAA.nn2 = <NN2_fqdn>:8020

- Add property dfs.client.failover.proxy.provider.<cluster - i.e HAA or HAB>

In cluster A

dfs.client.failover.proxy.provider.HAB = org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

In cluster B

dfs.client.failover.proxy.provider.HAA = org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

- Restart HDFS service.

Once complete you will be able to run the distcp command using the nameservice

Re: [FALCON] distcp failed firewall

New Contributor

@Ram Baskaran : it is not an issue from nameservice.

my question is concerning the ports need to be open to do distcp when the server port is 50070 (port 50070 is used by falcon during a submit of cluster entity)

I've already opened the port 50070 but it seems missing some thing because i've timeout when i did a distcp

Don't have an account?
Coming from Hortonworks? Activate your account here