Created 09-24-2018 06:50 AM
I am trying to run the distcp command in the secure cluster , My purpose is to move hdfs files from insecure cluster to secure cluster but i am getting errors .
hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true hdfs://<in-secure-hdfsnamenode>:8020/distsecure/f1.txt hdfs://<securenamenode>:8020/distdest/
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1537527981132_0012 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: <insecurenamenode>:8020, Ident: (HDFS_DELEGATION_TOKEN token 0 for hdfs) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:317) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155) at org.apache.hadoop.tools.DistCp.run(DistCp.java:128) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:462) Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1537527981132_0012 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: <insecurehdfs>:8020, Ident: (HDFS_DELEGATION_TOKEN token 0 for hdfs) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:272) at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:302)
I do not understand why in the error log i see failed to renew token for the insecure cluster .
also I have added ipc.client.fallback-to-simple-auth-allowed=true in the custom hdfs site. in the secure cluster .
Created 09-24-2018 06:56 AM
Can you try the distcp with this extra parameter;
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=remotenamenode1,remotenamenode2
Created 02-19-2019 12:37 PM
I got some error while RUNNING YARN-MapReduce example job.
How I can set up "token-renewal.exclude" parameter for YARN jobs?