distcp : Can't get Master Kerberos principal for use as renewer

AxelJ — Fri, 16 Sep 2022 10:23:38 GMT

Hi all,

We have just install a Kerberized cluster.

I would like to copy data from the former unsecured cluster (opendatarec) to the secured one (opendata) using distcp.

I run the following command on the secured cluster :

[hdfs@opendatafr01f ~]$ hadoop distcp webhdfs://opendatarecm01f.bi3.local:50070/toto webhdfs://opendatam01f.bi3.local:50070/tmp/

and get the following error :

mp/
16/06/06 17:46:55 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[webhdfs://opendatarecm01f.bi3.local:50070/toto], targetPath=webhdfs://opendatam01f.bi3.local:50070/tmp, targetPathExists=true, preserveRawXattrs=false, filtersFile='null'}
16/06/06 17:46:55 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/06/06 17:46:55 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/06/06 17:46:55 ERROR tools.DistCp: Exception encountered
java.io.IOException: Can't get Master Kerberos principal for use as renewer
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:133)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:164)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:372)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:172)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:123)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:436)

Any suggestion ?

Thanks

Re: distcp : Can't get Master Kerberos principal for use as renewer

Harsh J — Mon, 06 Jun 2016 16:22:22 GMT

On the host where you are invoking this command, do you have proper client
configs (gateway roles if you use CM) properly deployed?

Please also look at our DistCp matrix to learn how/where to invoke the
DIstCp command depending on your situation:
http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_distcp_data_cluster_migrate.html#concept_hfx_tqr_rp_section

Re: distcp : Can't get Master Kerberos principal for use as renewer

AxelJ — Tue, 07 Jun 2016 09:09:22 GMT

Hi Harsh

Thanks for your reply.

Yarn gateway was missing and adding them fixed the issue.

Re: distcp : Can't get Master Kerberos principal for use as renewer

Harsh J — Wed, 08 Jun 2016 04:27:22 GMT

Glad to hear @AxelJ! Feel free to mark the thread resolved so its easier for others with similar problems to locate your solution.

question Re: distcp : Can't get Master Kerberos principal for use as renewer in Archives of Support Questions (Read Only)

distcp : Can't get Master Kerberos principal for use as renewer

Re: distcp : Can't get Master Kerberos principal for use as renewer

Re: distcp : Can't get Master Kerberos principal for use as renewer

Re: distcp : Can't get Master Kerberos principal for use as renewer