Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn : Automatic clearing of filecache & usercache not kicking off

Yarn : Automatic clearing of filecache & usercache not kicking off

New Contributor

We are running a spark streaming job with yarn as resource manager, noticing that these two directories are getting filled up on the data nodes and we are running out of space when we only run for couple of min's

/tmp/hadoop/data/nm-local-dir/filecache

/tmp/hadoop/data/nm-local-dir/filecache

these directories are not getting cleared automatically , from my research found that this property need's to be set, yarn.nodemanager.localizer.cache.cleanup.interval-ms

Even after setting this up ..it's not automatically clearing out any help will be greatly appreciated

<configuration>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.aux-services</name>
    ~       <value>mapreduce_shuffle</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.resourcemanager.hostname</name>
    ~       <value>hdfs-name-node</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.resource.memory-mb</name>
    ~       <value>16384</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.resource.cpu-vcores</name>
    ~       <value>6</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.scheduler.maximum-allocation-mb</name>
    ~       <value>16384</value>
    ~   </property>
         <property>
    ~       <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
    ~       <value>3000</value>
    ~   </property>
    ~
    ~   <!-- Needs to be explicitly set as part of a workaround for YARN-367.
    ~      | If changing this property, you must also change the
    ~      | hadoop.tmp.dir property in hdfs-site.xml. This location must always
    ~      | be a subdirectory of the location specified in hadoop.tmp.dir. This
    ~      | affects all versions of Yarn 2.0.0 through 2.7.3+. -->
    ~   <property>
    ~       <name>yarn.nodemanager.local-dirs</name>
    ~       <value>file:///tmp/hadoop/data/nm-local-dir</value>
    ~   </property>
    ~
    ~ </configuration>
1 REPLY 1
Highlighted

Re: Yarn : Automatic clearing of filecache & usercache not kicking off

Hi @Aditya Mamidala,

You also need to set yarn.nodemanager.localizer.cache.target-size-mb along with yarn.nodemanager.localizer.cache.cleanup.interval-ms.

Also the resources are not cleaned up if they are being used by running containers.

More details given in below blog

https://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/