How to debug MapReduce job ?

[#1]

I am trying execute one MapReduce job in java but it is getting stuck at the middle and finally it was timed out. Below is the log

16/09/26 18:46:42 INFO mapreduce.Job: Running job: job_1474692614849_0070
16/09/26 18:46:50 INFO mapreduce.Job: Job job_1474692614849_0070 running in uber mode : false
16/09/26 18:46:50 INFO mapreduce.Job:  map 0% reduce 0%
16/09/26 18:47:01 INFO mapreduce.Job:  map 33% reduce 0%
16/09/26 18:52:19 INFO mapreduce.Job: Task Id : attempt_1474692614849_0070_m_000000_0, Status : FAILED
AttemptID:attempt_1474692614849_0070_m_000000_0 Timed out after 300 secs
16/09/26 18:52:20 INFO mapreduce.Job:  map 0% reduce 0%
16/09/26 18:52:30 INFO mapreduce.Job:  map 33% reduce 0%
16/09/26 18:57:49 INFO mapreduce.Job: Task Id : attempt_1474692614849_0070_m_000000_1, Status : FAILED
AttemptID:attempt_1474692614849_0070_m_000000_1 Timed out after 300 secs
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143


16/09/26 18:57:50 INFO mapreduce.Job:  map 0% reduce 0%
16/09/26 18:57:59 INFO mapreduce.Job:  map 33% reduce 0%
16/09/26 19:03:19 INFO mapreduce.Job: Task Id : attempt_1474692614849_0070_m_000000_2, Status : FAILED
AttemptID:attempt_1474692614849_0070_m_000000_2 Timed out after 300 secs
16/09/26 19:03:20 INFO mapreduce.Job:  map 0% reduce 0%
16/09/26 19:03:31 INFO mapreduce.Job:  map 33% reduce 0%
16/09/26 19:08:50 INFO mapreduce.Job:  map 100% reduce 100%
16/09/26 19:08:50 INFO mapreduce.Job: Job job_1474692614849_0070 failed with state FAILED due to: Task failed task_1474692614849_0070_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0


16/09/26 19:08:50 INFO mapreduce.Job: Counters: 13
        Job Counters
                Failed map tasks=4
                Killed reduce tasks=1
                Launched map tasks=4
                Other local map tasks=3
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=1311303
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=1311303
                Total time spent by all reduce tasks (ms)=0
                Total vcore-seconds taken by all map tasks=1311303
                Total vcore-seconds taken by all reduce tasks=0
                Total megabyte-seconds taken by all map tasks=2685548544
                Total megabyte-seconds taken by all reduce tasks=0

Is there any way to debug this mapreduce job ? Please help.

[#2 · Accepted solution]

@Arkaprova Saha

Can you please login to resource manager UI(http://<RM-hostname-or-ip:8088) and search for job_1474692614849_0070.

You can see application master/mapper/reducer logs to see exactly what has happened and why your job got timed out.

Hope this information helps.

[#3]

@Kuldeep Kulkarni Thanks for your reply and I will check.