Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark: Spark not using the all the executors configured

Spark: Spark not using the all the executors configured

Explorer

We are running a spark streaming application it has batches queued up ..but it's not using all the executors that were configured to it ..

 

SparkUIConfiguration.PNG

 

it's configured to use 24 executors but actually it's only using 16 and batches are getting queued up

 

SparkUIExecutorsUsed.PNG

 

how can we make it use all the 24 executors and not let it queue batches up

2 REPLIES 2

Re: Spark: Spark not using the all the executors configured

Expert Contributor

You will need to ensure there is enough YARN resources available to run all 24 executors.  It's possible that it only has enough resources to run 16 executors at that time.

Highlighted

Re: Spark: Spark not using the all the executors configured

Explorer

@hubbarja it might be but i'm not sure why yarn is not reflecting the coorect executor count configured in yarn-site.xml 

 

Yarn-ExecutorCount.PNG

 

it shows only 16Gb memory but 22GB is allocated via yarn-site.xml

<configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hdfs-name-node</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>22528</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>7</value>
    </property>

    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>22528</value>
    </property>

    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value>
    </property>

    <property>
        <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
        <value>500</value>
    </property>

    <property>
         <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
         <value>512</value>
    </property>

    </configuration>