Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

distcp insecure to secure hadoop cluster.

Explorer
## Trying to distcp from insecure hadoop cluster(2.3.4) to secure(2.4.2) hadoop cluster and it's failing with below error "Invalid arguments: SIMPLE authentication is not enabled"

## Can any one confirm, is it possible to distcp from insecure to secure hadoop cluster as i have tried with secure to insecure and it's working fine. And also below link is the hdp distcp matrix and where in "HDP2.x" only from secure to insecure hdfs was successfully but no information about insecure to secure.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Sys_Admin_Guides/content/ref-cfb69f75-d0...

## If anyone has tested insecure to secure, can you kindly confirm this. And is their any configuration required for insecure to secure distcp.

## And, while doing insecure to secure distcp, it's failing with "SIMPLE authentication is not enabled". I hope it's saying about target cluster(secure) and is their any way to enable simple authentication on secure cluster.

## And also below distcp throws "Invalid arguments" error, any idea why distcp is throwing this error, am i missing anything in the distcp.
===========================
hdfs@master02:~> hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true hdfs://HDP23:8020/test01.txt hdfs://HDP24:8020/
17/04/05 00:09:28 ERROR tools.DistCp: Invalid arguments:
org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2118)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1315)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
        at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:116)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
        at org.apache.hadoop.ipc.Client.call(Client.java:1427)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
        ... 9 more
Invalid arguments: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
usage: distcp OPTIONS [source_path...] <target_path>
========================================
9 REPLIES 9

Super Guru
@Turing nix

Set the following value in your core-site.xml and try again:

<property> 
  <name>ipc.client.fallback-to-simple-auth-allowed</name>
  <value>true</value>  
</property>

or just do it at the command line that way you don't change the security settings:

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo

Explorer

@mqureshi

Thanks for your inputs, but you had reviewed my properly i had already set the fallback property in distcp command.

Super Guru

@Turing nix

My apologies and I also missed the part that your destination is secure. You need to run command from the secure cluster and that means the user running the command is able to write to this secure cluster. Allowing what you are trying to do would be a security breach. See the following distcp matrix:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_Sys_Admin_Guides/content/distcp_data_cop...

@Turing nix- Below post talks about Running distcp between two cluster: One Kerberized and the other is not:

https://community.hortonworks.com/questions/294/running-distcp-between-two-cluster-one-kerberized.ht...

Hope this helps!

Explorer

@Namit Maheshwari

I have followed that article but still i was facing the same issue.

Explorer

@mqureshi @Namit Maheshwari @David Streever @Neeraj Sabharwal

Finally, i was able to below distcp from insecure to secure cluster by running the below distcp command from secured cluster.

## hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://<insecure_hdp>/test01.txt hdfs://<secure_hdp>/user/hdfs

And when i do below distcp from insecure to secure hadoop cluster on a insecure cluster, i was getting error "SIMPLE authentication is not enabled."

## hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://<insecure_hdp>/test01.txt hdfs://<secure_hdp>/user/hdfs

Can anyone tell, whats the difference in running the same distcp command in secure and insecure cluster, as on insecure cluster it fails.

Super Guru
@Turing nix

I missed this earlier as I stated in my comment. If you are able to run this from insecure cluster then it means your secure cluster is not really secured. Once you are on secured cluster and run this command after kinit, then there is no security breach. This is how it is supposed to work.

Super Guru

@Turing nix

My apologies and I also missed the part that your destination is secure. You need to run command from the secure cluster and that means the user running the command is able to write to this secure cluster. Allowing what you are trying to do would be a security breach. See the following distcp matrix:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_Sys_Admin_Guides/content/distcp_data_cop...

@Turing nix - For Distcp the general rule-of-thumb is that if one cluster is secure and the other is not secure, DistCp command should be ran from the secure cluster:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_Sys_Admin_Guides/content/distcp_and_secu...

So, if you are able to run distcp from a secure cluster to an unsecure cluster than there should be no issues.