YARN user cache limit

[#1]

Hi,

  I would like to know if there is some way to restrict how much disks space can a YARN user use in the YARN's NodeManager user cache. I would like to avoid to fill up he entire disk by some accident by only one user. 

Is there a way to set lets say that every user can have X amount of GB for usercache in YARN? If not, can I somehow instruct YARN to use different folder (drive) for non-production users and thus avoid the consumption of all free space? 

Thanks

[#2]

@Tomas79 - 

I am not sure, what you are asking is possible. But you can control the size of the same using below properties.

We could set below parameters to restrict the size of local directories under NM and trigger DeletionService when the limit is reached.

# yarn.nodemanager.delete.thread-count:
# yarn.nodemanager.localizer.cache.target-size-mb
# yarn.nodemanager.localizer.cache.cleanup.interval-ms

Details under: https://blog.cloudera.com/resource-localization-in-yarn-deep-dive/

The yarn.nodemanager.localizer.cache.target-size-mb property defines decides the maximum disk space to be used for localizing resources. Once the total disk size of the cache exceeds the value defined in this property the deletion service will try to remove files which are not used by any running containers.

The yarn.nodemanager.localizer.cache.cleanup.interval-ms: defines this interval for the delete the unused resources if total cache size exceeds the configured max-size. Unused resources are those resources which are not referenced by any running container.