Support Questions
Find answers, ask questions, and share your expertise

Few executers running on Yarn never stop processing and app freeze down

New Contributor

I have a job which is calling a spring boot application end points and getting some data from it.

I have roughly 400,000,00 data points which are being sent to end point using Map function. 

I am running application on Yarn and I have assigned 300+ executors each having 8 gigs memory. 

Everything runs smoothly, I can see requests coming and going on spring boot application. 

All executors finish their job in time but a couple of them ( mostly 1 or sometimes 2-5 ) keep on waiting and doing absolutely nothing. And that last step never finishes. 

Their thread dump shows they are waiting for some threads. 

I have tried to turn on dynamic allocation but still having the same problem. 


What could be the issue?




New Contributor

last executer remaining and never finishinglast executer remaining and never finishingThread dump of last remaining executerThread dump of last remaining executer

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.