question Event Correlation and Anomaly Detection in metron in Archives of Support Questions (Read Only)

Event Correlation and Anomaly Detection in metron

alizadeh_uut1 — Sun, 02 Oct 2016 18:07:08 GMT

hello to all,

I have reviewed metron docs and it's been indicated (for many times) that telemetry correlation and anomaly detection are two of metron main tasks.

Now i need to know which components do these tasks. I'm interested to see the source code doing correlation & anomaly detection.

Has anyone any idea?does anybody know where can I find them?

Thanks in advance.

Re: Event Correlation and Anomaly Detection in metron

TimothySpann — Mon, 10 Oct 2016 22:42:50 GMT

In-depth video is helpful: https://www.youtube.com/watch?v=5a3HywxY2kQ

https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture#MetronArchitecture-Step1-TelemetryEventBuffer

Adding a new event source

https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source

Look through the source:

https://github.com/apache/incubator-metron/tree/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-platform/metron-enrichment

Profiling Behavior of Entities

https://github.com/apache/incubator-metron/tree/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-analytics/metron-profiler

https://cwiki.apache.org/confluence/display/METRON/Metron+Wiki

Models are stored and data is compared against it.

Re: Event Correlation and Anomaly Detection in metron

sball — Sat, 22 Oct 2016 03:16:26 GMT

There are a variety of meanings of correlation in Metron.

One means is adding correlation keys in the enrichment process, which then allow you to view events together once they are indexed. So you can correlate events by adding a common search key you can then pivot on in Kibana. This is a great means of investigating correlations between alerts and events.

For a more statistical approach to correlation, you will want to look into the profiler which maintains windows of data, which can then be used to correlate time series data using, for example, arima in a model managed by the model as a service infrastructure. This area of Metron is growing quite fast at the moment. I would suggest also looking at the Stellar statistics functions which can be used to build simple anomaly based models as well. It's also easy enough to add functions to Stellar if you want to extend the functionality.

Re: Event Correlation and Anomaly Detection in metron

alizadeh_uut1 — Mon, 31 Oct 2016 14:51:25 GMT

@Simon Elliston Ball @Timothy Spann

thank you both for your helpful answers, actually it took me a while to go through your links, but now I know what I needed.

Re: Event Correlation and Anomaly Detection in metron

abras — Wed, 23 May 2018 16:47:27 GMT

Could you elaborate on your findings please ? How can I trigger a complex alarm involving simple alarms from different logs ? Is Stellar of any help for it ?