About john_absher

john_absher · ‎04-10-2018

@Gour Saha Pulling the logs from the Node manager... No errors of any kind Application finishes, it says it's uploading the logs for each container, and then moves on to the next thing. Only thing that may be relevant is Current good log dirs are /mnt/resource/hadoop/yarn/log which is I think referring to the local log directory. There is also (Slf4jLog.java:info(67)) - Aliases are enabled Nothing else of note.

john_absher · ‎04-06-2018

@Gour Saha I've set yarn.nodemanager.remote-app-log-dir=/app-logs and then set the owner of /app-logs to yarn:hadoop and still the problem remains (obviously now I'm checking /app-logs instead of /tmp for log files) The cluster is not secure - Kerberos is disabled.

john_absher · ‎04-05-2018

I'm trying to view aggregated logs - everything seems to be set up correctly, and it appears logs are aggregating without issue. However, they're. I have yarn.log-aggregation-enable set to true and yarn.nodemanager.remote-app-log-dir set to /tmp /tmp is owned by hdfs and is accessible to the hadoop group I have yarn.log-aggregation.retain-seconds is set to 2419200. Once the job finished I don't see the directory /tmp/logs (or /tmp/{user} or any other new directory in /tmp) in hdfs. I am unable to view the job logs through Yarn UI. Running yarn logs -applicationId {application_id} returns no results. When I check on the job status in the Yarn UI, it shows Log Aggregation Status: Succeeded. Resource manager logs are available and show no errors. Any ideas as to where these logs are disappearing to? We're running hadoop 2.5.3

Online	Offline
Last Visited	‎04-10-2018 09:06 PM

Member Since	‎04-05-2018 10:04 PM
Last Visited	‎04-10-2018 09:06 PM
Posts	3

Cloudera Community

Re: Logs disappearing after Yarn Log Aggregation

Re: Logs disappearing after Yarn Log Aggregation

Logs disappearing after Yarn Log Aggregation