Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Event Correlation and Anomaly Detection in metron

Solved Go to solution

Event Correlation and Anomaly Detection in metron

Contributor

hello to all,

I have reviewed metron docs and it's been indicated (for many times) that telemetry correlation and anomaly detection are two of metron main tasks.

Now i need to know which components do these tasks. I'm interested to see the source code doing correlation & anomaly detection.

Has anyone any idea?does anybody know where can I find them?

Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Event Correlation and Anomaly Detection in metron

Guru

There are a variety of meanings of correlation in Metron.

One means is adding correlation keys in the enrichment process, which then allow you to view events together once they are indexed. So you can correlate events by adding a common search key you can then pivot on in Kibana. This is a great means of investigating correlations between alerts and events.

For a more statistical approach to correlation, you will want to look into the profiler which maintains windows of data, which can then be used to correlate time series data using, for example, arima in a model managed by the model as a service infrastructure. This area of Metron is growing quite fast at the moment. I would suggest also looking at the Stellar statistics functions which can be used to build simple anomaly based models as well. It's also easy enough to add functions to Stellar if you want to extend the functionality.

4 REPLIES 4

Re: Event Correlation and Anomaly Detection in metron

Guru

There are a variety of meanings of correlation in Metron.

One means is adding correlation keys in the enrichment process, which then allow you to view events together once they are indexed. So you can correlate events by adding a common search key you can then pivot on in Kibana. This is a great means of investigating correlations between alerts and events.

For a more statistical approach to correlation, you will want to look into the profiler which maintains windows of data, which can then be used to correlate time series data using, for example, arima in a model managed by the model as a service infrastructure. This area of Metron is growing quite fast at the moment. I would suggest also looking at the Stellar statistics functions which can be used to build simple anomaly based models as well. It's also easy enough to add functions to Stellar if you want to extend the functionality.

Re: Event Correlation and Anomaly Detection in metron

Contributor

@Simon Elliston Ball @Timothy Spann

thank you both for your helpful answers, actually it took me a while to go through your links, but now I know what I needed.

Re: Event Correlation and Anomaly Detection in metron

New Contributor

Could you elaborate on your findings please ? How can I trigger a complex alarm involving simple alarms from different logs ? Is Stellar of any help for it ?

Don't have an account?
Coming from Hortonworks? Activate your account here