Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

copy stuff with distcp from insecure cluster to secure cluster. Please help to me.

copy stuff with distcp from insecure cluster to secure cluster. Please help to me.

New Contributor

Unable to copy data from non kerberos cluster to kerberized cluster, getting below error

18/04/20 02:20:15 ERROR tools.DistCp: Invalid arguments: org.apache.hadoop.security.AccessControlException: Authentication required at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:457) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:113) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:738) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:582) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:612) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:608) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:987) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1003) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1443) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:227) at org.apache.hadoop.tools.DistCp.run(DistCp.java:118) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:462) Invalid arguments: Authentication required usage: distcp OPTIONS [source_path...] <target_path>

1 REPLY 1

Re: copy stuff with distcp from insecure cluster to secure cluster. Please help to me.

Mentor

@venkata ramireddy

You can use DistCp and WebHDFS to copy data between a secure cluster and an insecure cluster by doing the following:

Set ipc.client.fallback-to-simple-auth-allowed to true in core-site.xml on the secure cluster side:

<property> 
  <name>ipc.client.fallback-to-simple-auth-allowed</name>
  <value>true</value> 
</property>

Use commands such as the following from the secure cluster side only:

distcp webhdfs://insecureCluster webhdfs://secureCluster 
distcp webhdfs://secureCluster webhdfs://insecureCluster

I have found that using webhdfs instead of hdfs for the nonsecure host gets around this error by setting ipc.client.fallback-to-simple-auth-allowed=true on the console while running DISTCP

"hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo"

Hope that helps