Support Questions
Find answers, ask questions, and share your expertise

Distcp between Secured HA cluster to Non Secured HA cluster

Highlighted

Distcp between Secured HA cluster to Non Secured HA cluster

Team,

We have set up two clusters, one non secure HA cluster and other one kerberos enabled HA cluster. We want to copy the data from Non secure HA to secure HA cluster using distcp. We run the following command from Kerberos enabled HA cluster:

# hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true hdfs://non-secure-cluster-ID/user/mazin/test-file hdfs://secure-cluster-ID/user/mazin/

we get following error:

On client datanode which is in secured cluster - hadoop.log

java.io.EOFException: End of File Exception between local host is: "datanode1006.example.com/192.0.0.248"; destination host is: "namenode2001.example.com":8020; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException

On non-secure namenode2001 I see following error:

2017-11-10 03:28:36,469 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client 192.0.0.248 threw exception [java.lang.IllegalArgumentException: Illegal principal name mazin@EXAMPLE.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to mazin@EXAMPLE.COM] java.lang.IllegalArgumentException: Illegal principal name mazin@EXAMPLE.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to mazin@EXAMPLE.COM at org.apache.hadoop.security.User.<init>(User.java:50) at org.apache.hadoop.security.User.<init>(User.java:43) at org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1178) at org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1162) at org.apache.hadoop.util.ProtoUtil.getUgi(ProtoUtil.java:138) at org.apache.hadoop.util.ProtoUtil.getUgi(ProtoUtil.java:120) at org.apache.hadoop.ipc.Server$Connection.processConnectionContext(Server.java:1662) at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:1891) at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:1767) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1531) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:762) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:636) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:607) Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to mazin@EXAMPLE.COM at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.User.<init>(User.java:48) ... 12 more

Above error hints on hadoop.security.auth_to_local entry in core-site.xml, since this is non secure cluster why do we need auth_to_local_entry setting.

Note: we want to use hdfs:// to copy the file and not webhdfs://

Thanks in advance.

Mazin

1 REPLY 1
Highlighted

Re: Distcp between Secured HA cluster to Non Secured HA cluster

This issue got fixed after the changed the hadoop.security.auth_to_local from "DEFAULT" to "" on non secured core-site.xml.

hadoop.security.auth_to_local = ""