Running distcp between two cluster: One Kerberized and the other is not

[#1]

hadoop distcp -i -log /tmp/ hdfs://xxx:8020/apps/yyyy hdfs://xxx_cid/tmp/

In this case the "xxx" is the "un-secure" cluster, while "xxx_cid" in the secure cluster.

We are launching the job from the Kerberos cluster, with the appropriate kinit for the user and getting the following error:

java.io.IOException: Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.; Host Details : local host is: "xxx/10.x.x.x"; destination host is: "xxx":8020;

...

Caused by: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.

I thought by launching the job from the secure cluster, that we could avoid any access issues. But it appears that the processes are kicked off from the "source" cluster. In this case, that's the insecure cluster.

Idea's on getting around this?

[#2]

1. getting below error after running the command "hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo"
2. java.io.EOFException:End of FileException between local host is***; destination host is:***;
3. please suggest

[#3]

非安全集群被阻止rpc通信，使用webhdfs协议，hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://nn1:50070/foo/bar hdfs://nn2:8020/bar/foo

[#4]

getting below error after running the command "hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo"

java.io.EOFException: End of File Exception between local host is *** ; destination host is:***;


please suggest

[#5]

I have found that using webhdfs instead of hdfs for the nonsecure host gets around this error, e.g.,

"hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo"

[#6]

can you try this

hadoop distcp –D ipc.client.falback-tosimple-auth-allowed=true webhdfs://insecureCluster webhdfs://secureCluster

[#7]

We had similar issue.

Most likely you have following property in /etc/hadoop/conf/hdfs-site.xml:

<property>
  <name>dfs.namenode.acls.enabled</name>
  <value>true</value>
  </property

Remove this property or set it to "false". It should help.

[#8]

Am doing distcp from insecure to secure hadop cluster and am getting error "SIMPLE authentication isnot enabled". Can any one suggest.

hdfs@master02:~> hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true hdfs://HDP23:8020/test01.txt hdfs://HDP24:8020/

17/04/0500:09:28 ERROR tools.DistCp:Invalid arguments:org.apache.hadoop.security.AccessControlException: SIMPLE authentication isnot enabled.Available:[TOKEN, KERBEROS]

[#9]

LiteBlue is the U.S. Government website this is intended for authorized use only by Postal Service employees