question Re: /hadoop/yarn/local/usercache/hive/filecache/ growing large in Support Questions

/hadoop/yarn/local/usercache/hive/filecache/ growing large

MikeB — Mon, 25 Oct 2021 18:59:46 GMT

I have 5 Hadoop/HDFS nodes running RHEL 7.9, Ambari 2.7.5.0, HDP 3.1.5.

/hadoop/yarn/local/usercache/hive/filecache/ on all of the Hadoop nodes is growing very large and I want to decrease its size.

Is there a YARN setting I can change to lower the amount of filecache specified above? Do I need to manually delete directories that are X days old to clean this up?

Please advise. Thanks! Mike

Re: /hadoop/yarn/local/usercache/hive/filecache/ growing large

balajip — Tue, 26 Oct 2021 02:31:03 GMT

Hello @MikeB

As for automatic clean up not getting triggered, well it may be due to (or at-least related to) this unresolved bug reported on Yarn.
https://issues.apache.org/jira/browse/YARN-4540
You can reduce the below property value to lower the local file cache size.
yarn.nodemanager.localizer.cache.target-size-mb

Re: /hadoop/yarn/local/usercache/hive/filecache/ growing large

MikeB — Tue, 26 Oct 2021 20:30:22 GMT

@balajip Is there a preferred way to add that parameter (and the other one in the issue you linked)? Like, should I add it on via Ambari GUI -> YARN configs or edit the proper file (yarn-site.xml?)? If I edit files, which nodes do I edit the files on?

Thanks,

Mike

Re: /hadoop/yarn/local/usercache/hive/filecache/ growing large

balajip — Wed, 27 Oct 2021 10:29:24 GMT

@MikeB You can find/add the property in Ambari--> YARN--> Configs--> Advanced--> Customer yarn-site --> Add/find Property