Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark with Yarn how to configure yarn to use all vcores

Spark with Yarn how to configure yarn to use all vcores

Explorer

We are running a spark streaming job using yarn as cluster manager, i have dedicated 7 cores per node to each node ...via yarn-site.xml as shown in the pic below

 

YarnCore.PNG

 

when the job is running ..it's only using 2 vcores and 5 vcores are left alone and the job is slow with lot of batches queued up ..

how can we make it use all the 7 vcores ? that's available to it this is usage when running so that it speed's up our job

 

VcoresStack2.PNG

 

Would greatly appreciate if any of the experts in the community will help out as we are new to Yarn & Spark

 

 

 

 

 

 

1 REPLY 1
Highlighted

Re: Spark with Yarn how to configure yarn to use all vcores

Expert Contributor

From your first screen shot, you have already maxed yout your memory so you won't be able to allocate more yarn containers.  You may want to lower your spark memory settings or increase your cores per executor when submitting your spark application.

Don't have an account?
Coming from Hortonworks? Activate your account here