Support Questions

Find answers, ask questions, and share your expertise

hadoop distcp not running

avatar
Contributor

Hi,

I am using sandbox HDP-3.0.1.0 Ambari Version 2.7.1.0,

My hadoop distcp command is stuck, not sure what is happening;

[root@sandbox-hdp bin]# hadoop distcp hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/foodmart.db/currency hdfs://sandbox-hdp.hortonworks.com:8020/sandbox/tutorial-f
iles/20575/
ERROR: Tools helper /usr/hdp/3.0.1.0-187/hadoop/libexec/tools/hadoop-distcp.sh was not found.
23/09/22 13:37:35 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false
, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0, copyStrategy='uniformsize', preserveStatus=[BLOCKSIZE
], atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/foodmart.db/currency], targetPath=hdfs://s
andbox-hdp.hortonworks.com:8020/sandbox/tutorial-files/20575, filtersFile='null', blocksPerChunk=0, copyBufferSize=8192, verboseLog=false}, sourcePaths=[hdfs://sandbox-hdp.hortonworks.com:802
0/warehouse/tablespace/managed/hive/foodmart.db/currency], targetPathExists=true, preserveRawXattrsfalse
23/09/22 13:37:35 INFO client.RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.3:8050
23/09/22 13:37:35 INFO client.AHSProxy: Connecting to Application History server at sandbox-hdp.hortonworks.com/172.18.0.3:10200
23/09/22 13:37:35 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 3; dirCnt = 2
23/09/22 13:37:35 INFO tools.SimpleCopyListing: Build file listing completed.
23/09/22 13:37:35 INFO tools.DistCp: Number of paths in the copy list: 3
23/09/22 13:37:35 INFO tools.DistCp: Number of paths in the copy list: 3
23/09/22 13:37:35 INFO client.RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.3:8050
23/09/22 13:37:35 INFO client.AHSProxy: Connecting to Application History server at sandbox-hdp.hortonworks.com/172.18.0.3:10200
23/09/22 13:37:35 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/root/.staging/job_1695389098784_0002
23/09/22 13:37:35 INFO mapreduce.JobSubmitter: number of splits:2
23/09/22 13:37:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1695389098784_0002
23/09/22 13:37:36 INFO mapreduce.JobSubmitter: Executing with tokens: []
23/09/22 13:37:36 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.0.1.0-187/0/resource-types.xml
23/09/22 13:37:36 INFO impl.YarnClientImpl: Submitted application application_1695389098784_0002
23/09/22 13:37:36 INFO mapreduce.Job: The url to track the job: http://sandbox-hdp.hortonworks.com:8088/proxy/application_1695389098784_0002/                                                  
23/09/22 13:37:36 INFO tools.DistCp: DistCp job-id: job_1695389098784_0002
23/09/22 13:37:36 INFO mapreduce.Job: Running job: job_1695389098784_0002
23/09/22 13:42:28 INFO mapreduce.Job: Job job_1695389098784_0002 running in uber mode : false
23/09/22 13:42:28 INFO mapreduce.Job: map 0% reduce 0%
 

 

1 REPLY 1

avatar
Contributor

Hi Shashank,
Is the map reduce jobs are failing ?

If you are concerned about the ERROR:Tools helper /usr/hdp/3.0.1.0-187/hadoop/libexec/tools/hadoop-distcp.sh was not found

https://issues.apache.org/jira/browse/HADOOP-17042
This will not impact the DistcpJob