Support Questions
Find answers, ask questions, and share your expertise

Using Distcp between from no a kerberized cluster to a kerberized cluster

Expert Contributor

I need to move some data from a no kerberized cluster to a kerberized cluster, and I have to deal with a kerberization problems.

This is a first view of what I got on output of the distcp command:

#hadoop distcp hdfs://****hnn01:8020/tmp/elements hdfs://****hmn31:8020/tmp18/02/23 10:51:21 ERROR tools.DistCp: Invalid arguments:
java.lang.IllegalArgumentException: ****hmn31 at
        at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(
        at org.apache.hadoop.hdfs.DFSClient.<init>(
        at org.apache.hadoop.hdfs.DFSClient.<init>(
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(
        at org.apache.hadoop.fs.FileSystem.createFileSystem(
        at org.apache.hadoop.fs.FileSystem.access$200(
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(
        at org.apache.hadoop.fs.FileSystem$Cache.get(
        at org.apache.hadoop.fs.FileSystem.get(
        at org.apache.hadoop.fs.Path.getFileSystem(
Caused by: ****hmn31
        ... 16 more Invalid arguments: ****hmn31

also there is the hdp version problem : I would move data from 2.3 to 2.5 hdp.

any suggestion will be greatly appreciated



Did get chance to check below URL:

1. Added the following property to the core-site.xml on the client machine (secure cluster):


2. Added the following rule to the core-site.xml on the source cluster (non-secure)

RULE:[1:$1@$0](.*@<replace with REALM name>)s/@.*// 

After adding the above properties you should be able to perform distcp successfully.


Hope this helps you.