question Is there a best practice for logging within HDP in Archives of Support Questions (Read Only)

Is there a best practice for logging within HDP

jaideep_mehta — Thu, 31 Mar 2016 22:06:58 GMT

Re: Is there a best practice for logging within HDP

aervits — Fri, 01 Apr 2016 06:40:21 GMT

We use log4j for loggers, is there a specific service you need to configure logging for? here are some examples https://community.hortonworks.com/content/kbentry/2112/tip-prevent-solr-cloud-logs-from-filling-up-the-no.html

https://community.hortonworks.com/content/idea/208/make-knox-log-directory-configurable-via-ambari.html

Re: Is there a best practice for logging within HDP

jaideep_mehta — Tue, 03 May 2016 09:23:40 GMT

Thank @Artem Ervits some of these links are helpful, however let me clarify a little more...

As part of hadoop applications we will be generating logs at different levels by different components . You will typically have end user computing along with hadoop processes like hive pig MR etc. So what is the best practice there, given that the application may have some components outside the hadoop cluster, how can you get end to end view of what happened from logging for an application

Re: Is there a best practice for logging within HDP

ravi1 — Tue, 03 May 2016 09:34:14 GMT

You can use nifi to ingest logs into solr/elasticsearch and visualize using banana/kibana. You can also use nifi to ingest them into splunk. Since all hadoop component logs use log4j, it is easy to standardize log collection.

Re: Is there a best practice for logging within HDP

aervits — Tue, 03 May 2016 09:39:34 GMT

I happen to agree with @Ravi Mutyala nifi is great to bring logs into hadoop and track logs at the edge, there are built-in parsers and filters, you'll feel right at home with Nifi. Here are some nifi templates including working with logs https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Re: Is there a best practice for logging within HDP

jstraub — Tue, 03 May 2016 11:28:07 GMT

You might want to look at this => https://github.com/abajwa-hw/logsearch-service

Great tool for analyzing logs 🙂