Support Questions
Find answers, ask questions, and share your expertise

Few executers running on Yarn never stop processing and app freeze down

New Contributor

I have a job which is calling a spring boot application end points and getting some data from it.

I have roughly 400,000,00 data points which are being sent to end point using Map function. 

I am running application on Yarn and I have assigned 300+ executors each having 8 gigs memory. 

Everything runs smoothly, I can see requests coming and going on spring boot application. 

All executors finish their job in time but a couple of them ( mostly 1 or sometimes 2-5 ) keep on waiting and doing absolutely nothing. And that last step never finishes. 

Their thread dump shows they are waiting for some threads. 

I have tried to turn on dynamic allocation but still having the same problem. 

 

What could be the issue?

 

 

1 REPLY 1

New Contributor

last executer remaining and never finishinglast executer remaining and never finishingThread dump of last remaining executerThread dump of last remaining executer

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.