Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi 0.7 putsplunk processor to send log files to splunk for nifi alerts?

Solved Go to solution

Nifi 0.7 putsplunk processor to send log files to splunk for nifi alerts?

Contributor

Iam new to Alerting & Monitoring. If we want to setup alerts for nifi using splunk can we use putsplunk nifi processor or send log files directly to splunk?

Currently we are having applications use splunk where they send the log files directly to splunk for alerting. Which is the effective way to acheive monitoring and alerting for nifi using splunk? Thank you

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Nifi 0.7 putsplunk processor to send log files to splunk for nifi alerts?

You probably have a couple of options...

I don't think you want the same NiFi instance that is running your main dataflow, also using PutSplunk to monitor itself. If you had TailFile -> PutSplunk, where TailFile was tailing the same instance, it would potentially create a cycle where the more your tailed and sent to splunk, the more logs produced, the more you tailed, the more logs you produced, etc.

I would suggest a second NiFi instance (maybe even the MiNiFi Java agent) to monitor the logs of the main instance.

Another possibly simpler solution... configure the NiFi logback.xml to add a UDP/TCP appender that can send logs to Splunk. This way anything NiFi logs to nifi-app.log will get forwarded to Splunk.

Last option, slightly different than logging, NiFi has a concept called a ReportingTask that can be used send metrics and statistics to other systems. If that was the information you were interested in, you could implement a custom ReportingTask to send data to Splunk.

View solution in original post

4 REPLIES 4
Highlighted

Re: Nifi 0.7 putsplunk processor to send log files to splunk for nifi alerts?

You probably have a couple of options...

I don't think you want the same NiFi instance that is running your main dataflow, also using PutSplunk to monitor itself. If you had TailFile -> PutSplunk, where TailFile was tailing the same instance, it would potentially create a cycle where the more your tailed and sent to splunk, the more logs produced, the more you tailed, the more logs you produced, etc.

I would suggest a second NiFi instance (maybe even the MiNiFi Java agent) to monitor the logs of the main instance.

Another possibly simpler solution... configure the NiFi logback.xml to add a UDP/TCP appender that can send logs to Splunk. This way anything NiFi logs to nifi-app.log will get forwarded to Splunk.

Last option, slightly different than logging, NiFi has a concept called a ReportingTask that can be used send metrics and statistics to other systems. If that was the information you were interested in, you could implement a custom ReportingTask to send data to Splunk.

View solution in original post

Highlighted

Re: Nifi 0.7 putsplunk processor to send log files to splunk for nifi alerts?

Contributor

@Bryan Bende

Hi Bryan,

How to use logback.xml for logging into Splunk. Any suggestion or link or example would be helpful.

Thanks,

Highlighted

Re: Nifi 0.7 putsplunk processor to send log files to splunk for nifi alerts?

You would need to setup an appender in logback.xml that could talk to a Splunk input. One example would be a SocketAppender to send to a TCP input in Splunk.

https://github.com/bbende/jsonevent-producer/blob/master/src/main/resources/logback.xml#L8-L11

Highlighted

Re: Nifi 0.7 putsplunk processor to send log files to splunk for nifi alerts?

Contributor

@Bryan Thanks Bryn. I will try this and will let you know if this works or not.

Don't have an account?
Coming from Hortonworks? Activate your account here