java.lang.Exception: Container is not yet running. Current state is NEW

[#1]

All our jobs are failing and giving only one error :

16/12/12 01:55:49 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1481368045892_0922 is still in NEW

16/12/12 01:55:51 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1481368045892_0922 is still in NEW

16/12/12 01:55:53 INFO impl.YarnClientImpl: Application submission is not finished, submitted application application_1481368045892_0922 is still in NEW

When i checked the resource manager UI, i clicked on the logs and it shows as:

java.lang.Exception: Container is not yet running. Current state is NEW

We have recently done the HDP upgrade from 2.3.2.0 to 2.4.2.0

[#2]

can you verify your yarn classpath (yarn.application.classpath) has all the necessary jars/paths to jars

[#3 · Accepted solution]

Please check which queue is the job getting submitted to and how many jobs are running in the queue. Possible that the queue where the jobs are submitted queue does not have any resources and hence unable to allocate any resources to the job .

You can check the RM UI which will give you a snapshot of the actual state of the yarn resource allocation.

[#4]

Even i have been noticing this error. But the job did not fail in my case. i see that only around 1 or 2 mappers out of 20 or so are failing after waiting for 1800 secs. The resource manager attempts the failed mappers again and they run to success. How can i understand why those 1 or 2 mappers are failing? i could only see this message on the log

java.lang.Exception: Container is not yet running. Current state is NEW