Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

not able to distcp from insecure cluster to secure cluster

avatar

I am trying to run the distcp command in the secure cluster , My purpose is to move hdfs files from insecure cluster to secure cluster but i am getting errors .

hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true hdfs://<in-secure-hdfsnamenode>:8020/distsecure/f1.txt hdfs://<securenamenode>:8020/distdest/
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1537527981132_0012 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: <insecurenamenode>:8020, Ident: (HDFS_DELEGATION_TOKEN token 0 for hdfs)
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:317)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:128)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:462)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1537527981132_0012 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: <insecurehdfs>:8020, Ident: (HDFS_DELEGATION_TOKEN token 0 for hdfs)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:272)
        at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291)
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:302)

I do not understand why in the error log i see failed to renew token for the insecure cluster .

also I have added ipc.client.fallback-to-simple-auth-allowed=true in the custom hdfs site. in the secure cluster .

2 REPLIES 2

avatar

Hi @Anurag Mishra

Can you try the distcp with this extra parameter;

-Dmapreduce.job.hdfs-servers.token-renewal.exclude=remotenamenode1,remotenamenode2

avatar
Explorer

I got some error while RUNNING YARN-MapReduce example job.

How I can set up "token-renewal.exclude" parameter for YARN jobs?