Support Questions
Find answers, ask questions, and share your expertise

Tez job showing as "SUCCEEDED" but vertices are still in "RUNNING" state in DAGs

Expert Contributor

screen-shot-2017-04-24-at-53229-pm.png

Tez job is showing as SUCCEEDED but vertices are still in "RUNNING" state in DAGs. This caused one of the table data got wiped out. Still the Mappers and Reducers are in running state but the DAG finished. Also it moved empty file to the target path resulting in zero bytes.

[2017-04-21 18:09:57,116] {hive_hooks.py:130} INFO - INFO  : Map 1: 0(+2)/8	Map 12: 1/1	Map 5: 1/1	Map 8: 1/1	Map 9: 1/1	Reducer 11: 0/1	Reducer 3: 0/308	Reducer 4: 0/2	Reducer 7: 0/1
[2017-04-21 18:29:38,595] {hive_hooks.py:130} INFO - INFO  : Map 1: 0(+2)/8	Map 12: 1/1	Map 5: 1/1	Map 8: 1/1	Map 9: 1/1	Reducer 11: 0/1	Reducer 3: 0/308	Reducer 4: 0/2	Reducer 7: 0/1
[2017-04-21 18:29:38,595] {hive_hooks.py:130} INFO - INFO  :
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO  : Status: DAG finished successfully in 1560.07 seconds
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO  :
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - 
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO  : METHOD                         DURATION(ms)
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO  : parse                                    0
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO  : semanticAnalyze                          0
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO  : TezBuildDag                              0
[2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO  : TezSubmitToRunningDag               12,957
[2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - INFO  : TotalPrepTime                       53,372
[2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - 
[2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - INFO  : Loading data to table dw_insight.d_experiment from hdfs://dfs-nameservices/data/insight/dwuser/d_experiment/.hive-staging_hive_2017-04-21_18-02-31_741_4202396962843353301-4404/-ext-10000
[2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - INFO  : Table dw_insight.d_experiment stats: [numFiles=0, numRows=0, totalSize=0, rawDataSize=0]


This job has been running in production for many months and recently we hit this scenario. We couldn't find the root cause of this issue. Is it bug from Tez ?

0 REPLIES 0