Reply
Explorer
Posts: 25
Registered: ‎01-10-2017

Spark: Spark not using the all the executors configured

We are running a spark streaming application it has batches queued up ..but it's not using all the executors that were configured to it ..

 

SparkUIConfiguration.PNG

 

it's configured to use 24 executors but actually it's only using 16 and batches are getting queued up

 

SparkUIExecutorsUsed.PNG

 

how can we make it use all the 24 executors and not let it queue batches up

Cloudera Employee
Posts: 97
Registered: ‎05-10-2016

Re: Spark: Spark not using the all the executors configured

You will need to ensure there is enough YARN resources available to run all 24 executors.  It's possible that it only has enough resources to run 16 executors at that time.

Highlighted
Explorer
Posts: 25
Registered: ‎01-10-2017

Re: Spark: Spark not using the all the executors configured

@hubbarja it might be but i'm not sure why yarn is not reflecting the coorect executor count configured in yarn-site.xml 

 

Yarn-ExecutorCount.PNG

 

it shows only 16Gb memory but 22GB is allocated via yarn-site.xml

<configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hdfs-name-node</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>22528</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>7</value>
    </property>

    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>22528</value>
    </property>

    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value>
    </property>

    <property>
        <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
        <value>500</value>
    </property>

    <property>
         <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
         <value>512</value>
    </property>

    </configuration>