Support Questions
Find answers, ask questions, and share your expertise

Killed reducer on join two tables

Killed reducer on join two tables

Contributor

Mappers done successfully

60529-failed-reducer-operation.png

[serv03 err_import_log]$ yarn logs -containerId container_e144_1518080001967_7656_01_000030 | tail -80
18/02/14 09:33:53 INFO client.AHSProxy: Connecting to Application History server at serv02.kyivstar.ua/10.49.72.2:10200
18/02/14 09:33:53 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/02/14 09:33:53 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm1]
18/02/14 09:33:54 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
18/02/14 09:33:54 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
2006.408: [GC (Allocation Failure) [PSYoungGen: 638570K->3653K(914944K)] 6504932K->5870015K(7159808K), 0.0124072 secs] [Times: user=0.33 sys=0.01, real=0.01 secs]
2007.306: [GC (Allocation Failure) [PSYoungGen: 705422K->1462K(915968K)] 6571784K->5867824K(7160832K), 0.0166994 secs] [Times: user=0.39 sys=0.00, real=0.02 secs]
2018-02-13 19:22:53 Completed running task attempt: attempt_1518080001967_7656_1_08_000052_0
2018-02-13 19:22:53 Starting to run new task attempt: attempt_1518080001967_7656_1_09_000045_0
2340.619: [GC (Allocation Failure) [PSYoungGen: 567942K->6728K(917504K)] 6434305K->5873099K(7162368K), 0.0203524 secs] [Times: user=0.35 sys=0.01, real=0.02 secs]
9499.320: [GC (Allocation Failure) [PSYoungGen: 749955K->401K(918016K)] 6616326K->5871440K(7162880K), 0.0495368 secs] [Times: user=0.41 sys=0.01, real=0.05 secs]
17292.616: [GC (Allocation Failure) [PSYoungGen: 753823K->800K(917504K)] 6624862K->5871934K(7162368K), 0.0360513 secs] [Times: user=0.49 sys=0.01, real=0.04 secs]
24939.182: [GC (Allocation Failure) [PSYoungGen: 742754K->608K(918016K)] 6613888K->5871750K(7162880K), 0.0399950 secs] [Times: user=0.29 sys=0.00, real=0.04 secs]
33559.139: [GC (Allocation Failure) [PSYoungGen: 835116K->736K(878592K)] 6706259K->5871878K(7123456K), 0.0552696 secs] [Times: user=0.39 sys=0.01, real=0.06 secs]
41484.916: [GC (Allocation Failure) [PSYoungGen: 771470K->672K(862208K)] 6642613K->5871814K(7107072K), 0.0432570 secs] [Times: user=0.36 sys=0.01, real=0.04 secs]
49639.787: [GC (Allocation Failure) [PSYoungGen: 791903K->832K(825344K)] 6663046K->5871974K(7070208K), 0.0334711 secs] [Times: user=0.34 sys=0.01, real=0.03 secs]
Heap
 PSYoungGen      total 825344K, used 128507K [0x0000000787200000, 0x00000007bea80000, 0x00000007c0000000)
  eden space 824320K, 15% used [0x0000000787200000,0x00000007947d0e90,0x00000007b9700000)
    lgrp 0 space 119972K, 23% used [0x0000000787200000,0x0000000788e06fb8,0x000000078e729000)
    lgrp 1 space 704348K, 14% used [0x000000078e729000,0x00000007947d0e90,0x00000007b9700000)
  from space 1024K, 81% used [0x00000007be980000,0x00000007bea50000,0x00000007bea80000)
  to   space 10752K, 0% used [0x00000007bd580000,0x00000007bd580000,0x00000007be000000)
 ParOldGen       total 6244864K, used 5871142K [0x00000005c0000000, 0x000000073d280000, 0x0000000787200000)
  object space 6244864K, 94% used [0x00000005c0000000,0x0000000726589ae8,0x000000073d280000)
 Metaspace       used 39999K, capacity 40338K, committed 40704K, reserved 1085440K
  class space    used 4485K, capacity 4599K, committed 4608K, reserved 1048576K
****************************************
==================================================================================================
LogType:syslog_attempt_1518080001967_7656_1_09_000045_0
LogLastModifiedTime:Wed Feb 14 08:50:36 +0200 2018
LogLength:2110
LogContents:
2018-02-14 08:50:28,843 [ERROR] [TezChild] |tez.ReduceRecordProcessor|: Hit error while closing operators - failing tree
2018-02-14 08:50:28,845 [ERROR] [TezChild] |tez.TezProcessor|: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
        at org.apache.tez.runtime.InputReadyTracker$InputReadyMonitor.awaitCondition(InputReadyTracker.java:120)
        at org.apache.tez.runtime.InputReadyTracker.waitForAllInputsReady(InputReadyTracker.java:90)
        at org.apache.tez.runtime.api.impl.TezProcessorContextImpl.waitForAllInputsReady(TezProcessorContextImpl.java:116)
        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:118)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


End of LogType:syslog_attempt_1518080001967_7656_1_09_000045_0
****************************************************************************************************************


[serv03 err_import_log]$