17/06/14 09:58:07 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=true, overwrite=false, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=24, mapBandwidth=1, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://nn1:8020/dir/year=2017/month=05/day=14], targetPath=hdfs://nn2:8020/dir/day=20170514, targetPathExists=false, filtersFile='null'} 17/06/14 09:58:07 INFO client.AHSProxy: Connecting to Application History server at hs/IP.73:10200 17/06/14 09:58:14 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 71821; dirCnt = 1 17/06/14 09:58:14 INFO tools.SimpleCopyListing: Build file listing completed. 17/06/14 09:58:15 INFO tools.DistCp: Number of paths in the copy list: 71821 17/06/14 09:58:15 INFO tools.DistCp: Number of paths in the copy list: 71821 17/06/14 09:58:15 INFO client.AHSProxy: Connecting to Application History server at hs/IP.73:10200 17/06/14 09:58:15 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]... 17/06/14 09:58:15 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm1] 17/06/14 09:58:16 INFO mapreduce.JobSubmitter: number of splits:25 17/06/14 09:58:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1496319477519_0056 17/06/14 09:58:17 INFO impl.YarnClientImpl: Submitted application application_1496319477519_0056 17/06/14 09:58:17 INFO mapreduce.Job: The url to track the job: http://rm:8088/proxy/application_1496319477519_0056/ 17/06/14 09:58:17 INFO tools.DistCp: DistCp job-id: job_1496319477519_0056 17/06/14 09:58:17 INFO mapreduce.Job: Running job: job_1496319477519_0056 17/06/14 09:58:24 INFO mapreduce.Job: Job job_1496319477519_0056 running in uber mode : false 17/06/14 09:58:24 INFO mapreduce.Job: map 0% reduce 0% 17/06/14 09:58:36 INFO mapreduce.Job: map 1% reduce 0% 17/06/14 09:58:39 INFO mapreduce.Job: map 2% reduce 0% 17/06/14 09:58:46 INFO mapreduce.Job: map 3% reduce 0% 17/06/14 09:58:52 INFO mapreduce.Job: map 4% reduce 0% 17/06/14 09:59:01 INFO mapreduce.Job: map 5% reduce 0% 17/06/14 09:59:58 INFO mapreduce.Job: map 6% reduce 0% 17/06/14 10:00:50 INFO mapreduce.Job: map 7% reduce 0% 17/06/14 10:01:45 INFO mapreduce.Job: map 8% reduce 0% 17/06/14 10:02:39 INFO mapreduce.Job: map 9% reduce 0% 17/06/14 10:03:32 INFO mapreduce.Job: map 10% reduce 0% 17/06/14 10:04:26 INFO mapreduce.Job: map 11% reduce 0% 17/06/14 10:05:20 INFO mapreduce.Job: map 12% reduce 0% 17/06/14 10:06:12 INFO mapreduce.Job: map 13% reduce 0% 17/06/14 10:07:08 INFO mapreduce.Job: map 14% reduce 0% 17/06/14 10:07:59 INFO mapreduce.Job: map 15% reduce 0% 17/06/14 10:08:54 INFO mapreduce.Job: map 16% reduce 0% 17/06/14 10:09:55 INFO mapreduce.Job: map 17% reduce 0% 17/06/14 10:10:49 INFO mapreduce.Job: map 18% reduce 0% 17/06/14 10:11:44 INFO mapreduce.Job: map 19% reduce 0% 17/06/14 10:12:37 INFO mapreduce.Job: map 20% reduce 0% 17/06/14 10:13:31 INFO mapreduce.Job: map 21% reduce 0% 17/06/14 10:14:23 INFO mapreduce.Job: map 22% reduce 0% 17/06/14 10:15:17 INFO mapreduce.Job: map 23% reduce 0% 17/06/14 10:16:12 INFO mapreduce.Job: map 24% reduce 0% 17/06/14 10:17:06 INFO mapreduce.Job: map 25% reduce 0% 17/06/14 10:17:59 INFO mapreduce.Job: map 26% reduce 0% 17/06/14 10:18:53 INFO mapreduce.Job: map 27% reduce 0% 17/06/14 10:19:49 INFO mapreduce.Job: map 28% reduce 0% 17/06/14 10:20:44 INFO mapreduce.Job: map 29% reduce 0% 17/06/14 10:21:38 INFO mapreduce.Job: map 30% reduce 0% 17/06/14 10:22:32 INFO mapreduce.Job: map 31% reduce 0% 17/06/14 10:23:25 INFO mapreduce.Job: map 32% reduce 0% 17/06/14 10:24:19 INFO mapreduce.Job: map 33% reduce 0% 17/06/14 10:25:13 INFO mapreduce.Job: map 34% reduce 0% 17/06/14 10:26:05 INFO mapreduce.Job: map 35% reduce 0% 17/06/14 10:26:59 INFO mapreduce.Job: map 36% reduce 0% 17/06/14 10:27:54 INFO mapreduce.Job: map 37% reduce 0% 17/06/14 10:28:47 INFO mapreduce.Job: map 38% reduce 0% 17/06/14 10:29:42 INFO mapreduce.Job: map 39% reduce 0% 17/06/14 10:30:36 INFO mapreduce.Job: map 40% reduce 0% 17/06/14 10:31:32 INFO mapreduce.Job: map 41% reduce 0% 17/06/14 10:32:26 INFO mapreduce.Job: map 42% reduce 0% 17/06/14 10:33:20 INFO mapreduce.Job: map 43% reduce 0% 17/06/14 10:34:15 INFO mapreduce.Job: map 44% reduce 0% 17/06/14 10:35:10 INFO mapreduce.Job: map 45% reduce 0% 17/06/14 10:36:04 INFO mapreduce.Job: map 46% reduce 0% 17/06/14 10:37:00 INFO mapreduce.Job: map 47% reduce 0% 17/06/14 10:37:52 INFO mapreduce.Job: Task Id : attempt_1496319477519_0056_m_000020_0, Status : FAILED Error: java.io.IOException: File copy failed: hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz --> hdfs://nn2:8020/dir/day=20170514/file.gz at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:287) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:255) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz to hdfs://nn2:8020/dir/day=20170514/file.gz at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:283) ... 10 more Caused by: java.io.IOException: Mismatch in length of source:hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz and target:hdfs://nn2:8020/dir/day=20170514/.distcp.tmp.attempt_1496319477519_0056_m_000020_0 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 17/06/14 10:37:53 INFO mapreduce.Job: map 46% reduce 0% 17/06/14 10:38:03 INFO mapreduce.Job: map 48% reduce 0% 17/06/14 10:38:55 INFO mapreduce.Job: map 49% reduce 0% 17/06/14 10:39:08 INFO mapreduce.Job: Task Id : attempt_1496319477519_0056_m_000020_1, Status : FAILED Error: java.io.IOException: File copy failed: hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz --> hdfs://nn2:8020/dir/day=20170514/file.gz at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:287) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:255) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz to hdfs://nn2:8020/dir/day=20170514/file.gz at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:283) ... 10 more Caused by: java.io.IOException: Mismatch in length of source:hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz and target:hdfs://nn2:8020/dir/day=20170514/.distcp.tmp.attempt_1496319477519_0056_m_000020_1 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 17/06/14 10:39:09 INFO mapreduce.Job: map 47% reduce 0% 17/06/14 10:39:19 INFO mapreduce.Job: map 49% reduce 0% 17/06/14 10:39:27 INFO mapreduce.Job: Task Id : attempt_1496319477519_0056_m_000020_2, Status : FAILED Error: java.io.IOException: File copy failed: hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz --> hdfs://nn2:8020/dir/day=20170514/file.gz at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:287) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:255) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz to hdfs://nn2:8020/dir/day=20170514/file.gz at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:283) ... 10 more Caused by: java.io.IOException: Mismatch in length of source:hdfs://nn1:8020/dir/year=2017/month=05/day=14/file.gz and target:hdfs://nn2:8020/dir/day=20170514/.distcp.tmp.attempt_1496319477519_0056_m_000020_2 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 17/06/14 10:39:28 INFO mapreduce.Job: map 47% reduce 0% 17/06/14 10:39:35 INFO mapreduce.Job: map 48% reduce 0% 17/06/14 10:39:38 INFO mapreduce.Job: map 49% reduce 0% 17/06/14 10:39:51 INFO mapreduce.Job: map 50% reduce 0% 17/06/14 10:40:47 INFO mapreduce.Job: map 51% reduce 0% 17/06/14 10:41:45 INFO mapreduce.Job: map 52% reduce 0% 17/06/14 10:41:53 INFO mapreduce.Job: map 100% reduce 0% 17/06/14 10:41:55 INFO mapreduce.Job: Job job_1496319477519_0056 failed with state FAILED due to: Task failed task_1496319477519_0056_m_000020 Job failed as tasks failed. failedMaps:1 failedReduces:0 17/06/14 10:41:55 INFO mapreduce.Job: Counters: 35 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=155359 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=20489475 HDFS: Number of bytes written=20486067 HDFS: Number of read operations=125 HDFS: Number of large read operations=0 HDFS: Number of write operations=28 Job Counters Failed map tasks=4 Killed map tasks=23 Launched map tasks=28 Other local map tasks=28 Total time spent by all maps in occupied slots (ms)=62589844 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=62589844 Total vcore-milliseconds taken by all map tasks=62589844 Total megabyte-milliseconds taken by all map tasks=64092000256 Map-Reduce Framework Map input records=13 Map output records=0 Input split bytes=115 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=72 CPU time spent (ms)=3860 Physical memory (bytes) snapshot=340471808 Virtual memory (bytes) snapshot=5562052608 Total committed heap usage (bytes)=548929536 File Input Format Counters Bytes Read=3293 File Output Format Counters Bytes Written=0 org.apache.hadoop.tools.mapred.CopyMapper$Counter BYTESCOPIED=20486067 BYTESEXPECTED=20486067 COPY=13 17/06/14 10:41:55 ERROR tools.DistCp: Exception encountered java.io.IOException: DistCp failure: Job job_1496319477519_0056 has failed: Task failed task_1496319477519_0056_m_000020 Job failed as tasks failed. failedMaps:1 failedReduces:0 at org.apache.hadoop.tools.DistCp.waitForJobCompletion(DistCp.java:215) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:158) at org.apache.hadoop.tools.DistCp.run(DistCp.java:128) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:462)