Reply
Highlighted
Expert Contributor
Posts: 115
Registered: ‎08-07-2017

oozie launcher logs

[ Edited ]

Hello All,

 

By default oozie launcher logs (jobhistory logs)get stored in /tmp/logs location in HDFS. Is there any way to store those logs in local filesystem instead of HDFS?

 

Kindly suggest.

 

Thanks,

Priya

Posts: 1,754
Kudos: 371
Solutions: 279
Registered: ‎07-31-2013

Re: oozie launcher logs

Oozie Launchers run as YARN jobs (single map-task MR job with a dummy input) and therefore their scheduling and logging all follow standard YARN methods, i.e. logs from whichever NodeManager the container got to run on gets uploaded to HDFS as part of YARN's log aggregation step when the container execution is completed.

You can download these logs to local filesystem by running 'yarn logs -applicationId <app ID of launcher job>' for example, but they cannot be written to a single local filesystem as the logging itself is occurring in a distributed environment.

If you need local-like-access to HDFS files for your scripts/etc. to discover these logs, you can consider mounting HDFS via NFS: https://www.cloudera.com/documentation/enterprise/latest/topics/admin_hdfs_nfsgateway.html
Announcements