Support Questions

Find answers, ask questions, and share your expertise

not able to distcp from insecure cluster to secure cluster

avatar

I am trying to run the distcp command in the secure cluster , My purpose is to move hdfs files from insecure cluster to secure cluster but i am getting errors .

hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true hdfs://<in-secure-hdfsnamenode>:8020/distsecure/f1.txt hdfs://<securenamenode>:8020/distdest/
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1537527981132_0012 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: <insecurenamenode>:8020, Ident: (HDFS_DELEGATION_TOKEN token 0 for hdfs)
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:317)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:128)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:462)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1537527981132_0012 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: <insecurehdfs>:8020, Ident: (HDFS_DELEGATION_TOKEN token 0 for hdfs)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:272)
        at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291)
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:302)

I do not understand why in the error log i see failed to renew token for the insecure cluster .

also I have added ipc.client.fallback-to-simple-auth-allowed=true in the custom hdfs site. in the secure cluster .

2 REPLIES 2

avatar

Hi @Anurag Mishra

Can you try the distcp with this extra parameter;

-Dmapreduce.job.hdfs-servers.token-renewal.exclude=remotenamenode1,remotenamenode2

avatar
Explorer

I got some error while RUNNING YARN-MapReduce example job.

How I can set up "token-renewal.exclude" parameter for YARN jobs?