question Approach to collect logs from mappers in Archives of Support Questions (Read Only)

Approach to collect logs from mappers

dzianis_frydlia — Fri, 10 Jun 2016 16:58:03 GMT

I have a YARN map/reduce application.

In mapper I use log4j to log certain cases. After job exection is finished I want to analyze logs from ALL mappers. As there're a lot of mappers in my job, log analysis becomes rather painful job...

Is there a way to write log from mappers to some aggregated file to have all the record in one place? Or probaly there's an approach to combine log files from all mappers from a concrete job?

Re: Approach to collect logs from mappers

rajkumar_singh — Fri, 10 Jun 2016 17:01:59 GMT

if you are running mapreduce over yarn then enable yarn remote log aggregation(http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/), it will centralized all logging for you and you can perform your analysis over on aggregated logs.

Re: Approach to collect logs from mappers

bleonhardi — Fri, 10 Jun 2016 17:06:15 GMT

yeah it should be enabled by default though. You would get the log files through the yarn logs command line or you can use pig as well.

https://community.hortonworks.com/articles/33703/mining-tez-app-log-file-with-pig-script.html

Re: Approach to collect logs from mappers

dzianis_frydlia — Fri, 10 Jun 2016 17:08:14 GMT

Thank you @Rajkumar Singh!

You would only add that it's rather convenient for me to use the following approach:

yarn logs -applicationId application_1465548978834_0004 | grep my.foo.class > /home/user/log2.txt

With this command I can filter all the log entries for the class I want to analyze.