Reply
Expert Contributor
Posts: 112
Registered: ‎08-07-2017

oozie launcher logs

[ Edited ]

Hello All,

 

By default oozie launcher logs (jobhistory logs)get stored in /tmp/logs location in HDFS. Is there any way to store those logs in local filesystem instead of HDFS?

 

Kindly suggest.

 

Thanks,

Priya

Highlighted
Posts: 1,749
Kudos: 365
Solutions: 277
Registered: ‎07-31-2013

Re: oozie launcher logs

Oozie Launchers run as YARN jobs (single map-task MR job with a dummy input) and therefore their scheduling and logging all follow standard YARN methods, i.e. logs from whichever NodeManager the container got to run on gets uploaded to HDFS as part of YARN's log aggregation step when the container execution is completed.

You can download these logs to local filesystem by running 'yarn logs -applicationId <app ID of launcher job>' for example, but they cannot be written to a single local filesystem as the logging itself is occurring in a distributed environment.

If you need local-like-access to HDFS files for your scripts/etc. to discover these logs, you can consider mounting HDFS via NFS: https://www.cloudera.com/documentation/enterprise/latest/topics/admin_hdfs_nfsgateway.html
Announcements