Reply
Highlighted
Expert Contributor
Posts: 114
Registered: ‎08-07-2017

exitCode: 11, (reason: Max number of executor failures (24) reached)

 Hello All,

 

We are running spark application. And frequently it is getting failed. In the log I see below message.

 

exitCode: 11, (reason: Max number of executor failures (24) reached)

 

And executor is getting failed with below error.

 

Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_e14_15320282824
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

 

Is there any limit on no. of executor failures  for spark application? I have specified no. of executors as 12.I don't see such parameter in cloudera manager though. Please suggest. 

As per my understanding, due to less memory,executors are getting failed an donce it reaches the max. limit, application is getting killed. We need to increase executor memory in this case. Kindly help.

 

Thanks,

Priya

Announcements