Created 03-31-2016 03:06 PM
Created 05-03-2016 02:34 AM
You can use nifi to ingest logs into solr/elasticsearch and visualize using banana/kibana. You can also use nifi to ingest them into splunk. Since all hadoop component logs use log4j, it is easy to standardize log collection.
Created 03-31-2016 11:40 PM
We use log4j for loggers, is there a specific service you need to configure logging for? here are some examples https://community.hortonworks.com/content/kbentry/2112/tip-prevent-solr-cloud-logs-from-filling-up-t...
Created 05-03-2016 02:23 AM
Thank @Artem Ervits some of these links are helpful, however let me clarify a little more...
As part of hadoop applications we will be generating logs at different levels by different components . You will typically have end user computing along with hadoop processes like hive pig MR etc. So what is the best practice there, given that the application may have some components outside the hadoop cluster, how can you get end to end view of what happened from logging for an application
Created 05-03-2016 02:39 AM
I happen to agree with @Ravi Mutyala nifi is great to bring logs into hadoop and track logs at the edge, there are built-in parsers and filters, you'll feel right at home with Nifi. Here are some nifi templates including working with logs https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
Created 05-03-2016 02:34 AM
You can use nifi to ingest logs into solr/elasticsearch and visualize using banana/kibana. You can also use nifi to ingest them into splunk. Since all hadoop component logs use log4j, it is easy to standardize log collection.
Created 05-03-2016 04:28 AM
You might want to look at this => https://github.com/abajwa-hw/logsearch-service
Great tool for analyzing logs 🙂