Can we change yarn.nodemanager.log-dirs value from local to hdfs ?

[#1]

Team,

Actually there are multiple jobs running on our servers and during running jobs are creating more staging data in local /var/log/yarn/log dir. I understand it is because of container and yarn.nodemanager.log-dirs property.

We have 100GB for this location but still it is getting full, so is there anyway where we can redirect it to hdfs ?

Thanks in advance.

[#2]

You can add a comma separated list of local/log directories that go to different disks instead of a single folder there to avoid location getting full. It is not a good idea to try to write it to hdfs (even if its possible which I doubt)

[#3]

@Ravi Mutyala: Yes, I can do it but problem is whenever one of configured dirs will be 100% utilized imidiately we will keep on getting alerts email. And problem is we can not disable those alerts as we have to monitor.

[#4]

Generally not a good idea to use /var/log for yarn.nodemanager.log-dirs which are container logs. Typically, we direct these logs to all the Data mount points (like /grid/N/yarn/log). Same thing for yarn local (/gird/N/yarn/local). This can help with reducing all your IO going to your OS disk (where you typically have /var/log)

[#5]

@Ravi Mutyala: Is there any hortonwork doc or reference where this recomanation is given ?

[#6]

You can take a look at http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/

• yarn.nodemanager.local-dirs: This is a comma separated list of local-directories that one can configure to be used for copying files during localization. The idea behind allowing multiple directories is to use multiple disks for localization – it helps both fail-over (one/few disk(s) going bad doesn’t affect all containers) and load balancing (no single disk is bottlenecked with writes). Thus, individual directories should be configured if possible on different local disks.

You can follow same approach for container logs as well.

[#7]

@Ravi Mutyala: I checked and as I have give above we have this value comma separated for disk only.

So not sure what is going wrong and why it is getting filled up even for a single big job.

[#8]

Take a look at below properties. You can reduce the retention time of local logs.

yarn.nodemanager.log.retain-seconds: Time in seconds to retain user logs on the individual nodes if log aggregation is disabled. Default is 10800

yarn.nodemanager.log.deletion-threads-count: Determines the number of threads used by the NodeManagers to clean-up logs once the log-retention time is hit for local log files when aggregation is disabled.