The other thing which i noticed while testing the same in other cluster, distcp is getting stuck when trying to connect to application history server. 20/03/23 20:38:40 INFO client.AHSProxy: Connecting to Application History server at host/ipaddress:10200 ... View more

I absolutely agree. Initially i was testing distcp with two trusted clusters with the below command. Since it was getting stuck, did a simple test to copy within the cluster but still the same issue. (All the prerequisites are met to do a distcp) hadoop distcp hdfs://nn:8020/user/hdfs_home_eid/test.txt hdfs://nn:8020/tmp/   just as an FYI. Below is the link has similar issue:- but it dint help me to resolve the problem. http://people.apache.org/~liuml07/2017/07/05/DistCp-gets-stuck-with-build-listing/ ... View more

Yes, the cluster is kerberized, HWX, HDP 3.1.5. can't seem to find logs for the below operation. Below is the simple command:- hadoop distcp /user/home/test.txt /tmp/   20/03/23 18:16:59 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0, copyStrategy='uniformsize', preserveStatus=[BLOCKSIZE], atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[/user/home/test.txt], targetPath=/tmp, filtersFile='null', blocksPerChunk=0, copyBufferSize=8192, verboseLog=false, directWrite=false}, sourcePaths=[/user/home/test.txt], targetPathExists=true, preserveRawXattrsfalse 20/03/23 18:16:59 INFO client.AHSProxy: Connecting to Application History server at host:10200 20/03/23 18:16:59 INFO hdfs.DFSClient: Created token for eid: HDFS_DELEGATION_TOKEN owner=EID@Domian.COM, renewer=yarn, realUser=, issueDate=1585001819568, maxDate=1585606619568, sequenceNumber=44990, masterKeyId=161 on ha-hdfs:nn-ha 20/03/23 18:16:59 INFO kms.KMSClientProvider: New token created: (Kind: kms-dt, Service: kms://https@host:9393/kms, Ident: (kms-dt owner=Eid, renewer=yarn, realUser=, issueDate=1585001819728, maxDate=1585606619728, sequenceNumber=11938, masterKeyId=7)) 20/03/23 18:16:59 INFO security.TokenCache: Got dt for hdfs://nn-ha; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nn-ha, Ident: (token for eid: HDFS_DELEGATION_TOKEN owner=eid@AOINS.COM, renewer=yarn, realUser=, issueDate=1585001819568, maxDate=1585606619568, sequenceNumber=44990, masterKeyId=161) 20/03/23 18:16:59 INFO security.TokenCache: Got dt for hdfs://nn-ha; Kind: kms-dt, Service: kms://https@host:9393/kms, Ident: (kms-dt owner=eid, renewer=yarn, realUser=, issueDate=1585001819728, maxDate=1585606619728, sequenceNumber=11938, masterKeyId=7) 20/03/23 18:16:59 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0 20/03/23 18:16:59 INFO tools.SimpleCopyListing: Build file listing completed.   When i do a ctrl+C to close the long running distcp job abruptly gives me below exception:-   ERROR hdfs.DFSClient: Failed to close file: /user/home/.staging/_distcp-1688802777/fileList.seq_sorted.0 with inode: 56047223 org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/home/.staging/_distcp-1688802777/fileList.seq_sorted.0 (inode 56047223) Holder DFSClient_NONMAPREDUCE_1216352325_1 does not have any open files.   ERROR hdfs.DFSClient: Failed to close file: /user/home/.staging/_distcp-1688802777/fileList.seq_sorted.0.index with inode: 56047224 org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/home/.staging/_distcp-1688802777/fileList.seq_sorted.0.index (inode 56047224) Holder DFSClient_NONMAPREDUCE_1216352325_1 does not have any open files. ERROR tools.DistCp: Exception encountered java.nio.channels.ClosedChannelException ... View more