Reply
Explorer
Posts: 62
Registered: ‎01-22-2014

Distcp Error in Secure CDH4.3

Hi,

 

I am trying to a Hadoop Distcp between two kerberos enabled CDH4.3.0 clusters (version - Hadoop 2.0.0-cdh4.3.0)

 

When I use the command "hadoop distcp hdfs:cluster1:8020/user/test.txt hdfs://cluster2:8020/user" in the destination cluster, it works fine. But when I exec the command in the source cluster, I am getting the below error -

 

 

Copy failed: java.io.IOException: Failed on local exception: java.io.IOException: Response is null.; Host Details : local host is: "cluster1/10.96.82.149"; destination host is: "cluster2":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
at org.apache.hadoop.ipc.Client.call(Client.java:1229)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy9.getDelegationToken(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy9.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:783)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:783)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:868)
at org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:509)
at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:487)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:130)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:111)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:85)
at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1046)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
Caused by: java.io.IOException: Response is null.
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:941)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:836)

 

 


When I try to use "hadoop distcp hftp:cluster1:50070/user/test.txt hdfs://cluster2:8020/user" on either the source or the destination cluster, I am getting the below error, but I have not enabled the property  'Enable Authentication for HTTP Web-Consoles'

 

Please let me know what needs to be done to execute the Distcp command in the source cluster.

 

 

org.apache.hadoop.ipc.RemoteException(java.io.IOException): Security enabled but user not authenticated by filter

at org.apache.hadoop.ipc.RemoteException.valueOf(RemoteException.java:97)
at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.startElement(HftpFileSystem.java:425)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:501)
at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:179)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:377)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl$NSContentDriver.scanRootElementHook(XMLNSDocumentScannerImpl.java:626)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3104)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:922)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:464)
at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:475)
at org.apache.hadoop.hdfs.HftpFileSystem.getFileStatus(HftpFileSystem.java:504)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1378)
at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:636)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

 



Explorer
Posts: 23
Registered: ‎02-11-2014

Re: Distcp Error in Secure CDH4.3

Hi Arun,

 

I hope you have configured your cluster with hdfs-clusters.conf file, if not try to configure this file in each node of your cluster.

 

Along with this try to check with below things to work Distcp on cluster.

 

sudo chmod -R 777 /tmp
sudo chown -R user:user /tmp

 

where user is the username from which you are running the command.

 

Atlast try to set java opts properties properly in your cluster and make sure all nodes in your cluster must have same configuration.

 

Regards,
Chirag Patadia.
Highlighted
Posts: 1,640
Kudos: 314
Solutions: 254
Registered: ‎07-31-2013

Re: Distcp Error in Secure CDH4.3

Doing a DistCp between two secured CDH4 clusters would require a cross-realm trust. Do you have that setup? Do you use JDK7?

You do not have to use HFTP (although that will require cross realm trust over SPNEGO too).
Announcements