Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Distcp created yarn application in new state not moving to next stage (not allocating Resource)

Highlighted

Distcp created yarn application in new state not moving to next stage (not allocating Resource)

New Contributor

All :

Problem : Source cluster configured with Dual Interface eth0 reachable outside of cluster, eth1 only reachable inside the cluster not outside of cluster.

When i ran hadoop distcp to copy data from this source cluster to destination cluster, the distcp job creating yarn application id but not moving from new state to further.

Both source and target are keberiosed cluster with different kerberos relam

Source Kerberos is on eth1.

Log When Executing Hadoop Distcp Command:

17/05/15 06:28:19 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://l4driqap4485/tmp/mughil1], targetPath=hdfs://l4dripap002.hewitt.com/tmp/mughil, targetPathExists=true, filtersFile='null'} 17/05/15 06:28:19 INFO hdfs.DFSClient: Created token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847699902, maxDate=1495452499902, sequenceNumber=17772, masterKeyId=294 on 192.168.2.2:8020 17/05/15 06:28:19 INFO security.TokenCache: Got dt for hdfs://mughil1; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.2.2:8020, Ident: (token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847699902, maxDate=1495452499902, sequenceNumber=17772, masterKeyId=294) 17/05/15 06:28:20 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 1 17/05/15 06:28:20 INFO tools.SimpleCopyListing: Build file listing completed. 17/05/15 06:28:20 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb 17/05/15 06:28:20 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor 17/05/15 06:28:20 INFO tools.DistCp: Number of paths in the copy list: 1 17/05/15 06:28:20 INFO tools.DistCp: Number of paths in the copy list: 1 17/05/15 06:28:20 INFO hdfs.DFSClient: Created token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847700265, maxDate=1495452500265, sequenceNumber=18543131, masterKeyId=1582 on 10.80.34.2:8020 17/05/15 06:28:20 INFO security.TokenCache: Got dt for hdfs://mughil2; Kind: HDFS_DELEGATION_TOKEN, Service: 10.80.34.2:8020, Ident: (token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847700265, maxDate=1495452500265, sequenceNumber=18543131, masterKeyId=1582) 17/05/15 06:28:20 INFO hdfs.DFSClient: Created token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847700339, maxDate=1495452500339, sequenceNumber=17773, masterKeyId=294 on ha-hdfs:nameservice1 17/05/15 06:28:20 INFO security.TokenCache: Got dt for hdfs://nameservice1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847700339, maxDate=1495452500339, sequenceNumber=17773, masterKeyId=294) 17/05/15 06:28:20 INFO mapreduce.JobSubmitter: number of splits:1 17/05/15 06:28:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1493920152177_2063 17/05/15 06:28:20 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.2.2:8020, Ident: (token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847699902, maxDate=1495452499902, sequenceNumber=17772, masterKeyId=294) 17/05/15 06:28:20 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 10.80.34.2:8020, Ident: (token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847700265, maxDate=1495452500265, sequenceNumber=18543131, masterKeyId=1582) 17/05/15 06:28:20 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494847700339, maxDate=1495452500339, sequenceNumber=17773, masterKeyId=294) 17/05/15 06:28:22 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1493920152177_2063 is still in NEW 17/05/15 06:28:24 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1493920152177_2063 is still in NEW 17/05/15 06:28:26 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1493920152177_2063 is still in NEW

Yarn Log :

017-05-15 06:20:06,787 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:yarn/mughil1@XXX.XXX.COM (auth:KERBEROS) cause:org.apache.hadoop.net.ConnectTimeoutException: Call From l4driqap4484/192.168.2.1 to l4dripap002.hewitt.com:8020 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending local=/192.168.2.1:50848 remote=l4dripap002.hewitt.com/10.80.34.2:8020]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout 2017-05-15 06:20:06,788 WARN org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Unable to add the application to the delegation token renewer. java.io.IOException: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: 10.80.34.2:8020, Ident: (token for mughil: HDFS_DELEGATION_TOKEN owner=mughil@XXX.XXX.COM, renewer=yarn, realUser=, issueDate=1494846285498, maxDate=1495451085498, sequenceNumber=18542786, masterKeyId=1582) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:465) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:79) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:895) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:872) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.net.ConnectTimeoutException: Call From mughil1/192.168.2.1 to mughil.abcd.com:8020 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending local=/192.168.2.1:50848 remote=l4dripap002.hewitt.com/10.80.34.2:8020]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout at sun.reflect.GeneratedConstructorAccessor115.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy86.renewDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewDelegationToken(ClientNamenodeProtocolTranslatorPB.java:943) at sun.reflect.GeneratedMethodAccessor119.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy87.renewDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:1184) at org.apache.hadoop.security.token.Token.renew(Token.java:414) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:586) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:583) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:581) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:463) ...