Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Killed reducer on join two tables

Highlighted

Killed reducer on join two tables

Contributor

Mappers done successfully

60529-failed-reducer-operation.png

[serv03 err_import_log]$ yarn logs -containerId container_e144_1518080001967_7656_01_000030 | tail -80
18/02/14 09:33:53 INFO client.AHSProxy: Connecting to Application History server at serv02.kyivstar.ua/10.49.72.2:10200
18/02/14 09:33:53 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
18/02/14 09:33:53 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm1]
18/02/14 09:33:54 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
18/02/14 09:33:54 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
2006.408: [GC (Allocation Failure) [PSYoungGen: 638570K->3653K(914944K)] 6504932K->5870015K(7159808K), 0.0124072 secs] [Times: user=0.33 sys=0.01, real=0.01 secs]
2007.306: [GC (Allocation Failure) [PSYoungGen: 705422K->1462K(915968K)] 6571784K->5867824K(7160832K), 0.0166994 secs] [Times: user=0.39 sys=0.00, real=0.02 secs]
2018-02-13 19:22:53 Completed running task attempt: attempt_1518080001967_7656_1_08_000052_0
2018-02-13 19:22:53 Starting to run new task attempt: attempt_1518080001967_7656_1_09_000045_0
2340.619: [GC (Allocation Failure) [PSYoungGen: 567942K->6728K(917504K)] 6434305K->5873099K(7162368K), 0.0203524 secs] [Times: user=0.35 sys=0.01, real=0.02 secs]
9499.320: [GC (Allocation Failure) [PSYoungGen: 749955K->401K(918016K)] 6616326K->5871440K(7162880K), 0.0495368 secs] [Times: user=0.41 sys=0.01, real=0.05 secs]
17292.616: [GC (Allocation Failure) [PSYoungGen: 753823K->800K(917504K)] 6624862K->5871934K(7162368K), 0.0360513 secs] [Times: user=0.49 sys=0.01, real=0.04 secs]
24939.182: [GC (Allocation Failure) [PSYoungGen: 742754K->608K(918016K)] 6613888K->5871750K(7162880K), 0.0399950 secs] [Times: user=0.29 sys=0.00, real=0.04 secs]
33559.139: [GC (Allocation Failure) [PSYoungGen: 835116K->736K(878592K)] 6706259K->5871878K(7123456K), 0.0552696 secs] [Times: user=0.39 sys=0.01, real=0.06 secs]
41484.916: [GC (Allocation Failure) [PSYoungGen: 771470K->672K(862208K)] 6642613K->5871814K(7107072K), 0.0432570 secs] [Times: user=0.36 sys=0.01, real=0.04 secs]
49639.787: [GC (Allocation Failure) [PSYoungGen: 791903K->832K(825344K)] 6663046K->5871974K(7070208K), 0.0334711 secs] [Times: user=0.34 sys=0.01, real=0.03 secs]
Heap
 PSYoungGen      total 825344K, used 128507K [0x0000000787200000, 0x00000007bea80000, 0x00000007c0000000)
  eden space 824320K, 15% used [0x0000000787200000,0x00000007947d0e90,0x00000007b9700000)
    lgrp 0 space 119972K, 23% used [0x0000000787200000,0x0000000788e06fb8,0x000000078e729000)
    lgrp 1 space 704348K, 14% used [0x000000078e729000,0x00000007947d0e90,0x00000007b9700000)
  from space 1024K, 81% used [0x00000007be980000,0x00000007bea50000,0x00000007bea80000)
  to   space 10752K, 0% used [0x00000007bd580000,0x00000007bd580000,0x00000007be000000)
 ParOldGen       total 6244864K, used 5871142K [0x00000005c0000000, 0x000000073d280000, 0x0000000787200000)
  object space 6244864K, 94% used [0x00000005c0000000,0x0000000726589ae8,0x000000073d280000)
 Metaspace       used 39999K, capacity 40338K, committed 40704K, reserved 1085440K
  class space    used 4485K, capacity 4599K, committed 4608K, reserved 1048576K
****************************************
==================================================================================================
LogType:syslog_attempt_1518080001967_7656_1_09_000045_0
LogLastModifiedTime:Wed Feb 14 08:50:36 +0200 2018
LogLength:2110
LogContents:
2018-02-14 08:50:28,843 [ERROR] [TezChild] |tez.ReduceRecordProcessor|: Hit error while closing operators - failing tree
2018-02-14 08:50:28,845 [ERROR] [TezChild] |tez.TezProcessor|: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
        at org.apache.tez.runtime.InputReadyTracker$InputReadyMonitor.awaitCondition(InputReadyTracker.java:120)
        at org.apache.tez.runtime.InputReadyTracker.waitForAllInputsReady(InputReadyTracker.java:90)
        at org.apache.tez.runtime.api.impl.TezProcessorContextImpl.waitForAllInputsReady(TezProcessorContextImpl.java:116)
        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:118)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


End of LogType:syslog_attempt_1518080001967_7656_1_09_000045_0
****************************************************************************************************************


[serv03 err_import_log]$