Event Correlation and Anomaly Detection in metron

[#1]

hello to all,

I have reviewed metron docs and it's been indicated (for many times) that telemetry correlation and anomaly detection are two of metron main tasks.

Now i need to know which components do these tasks. I'm interested to see the source code doing correlation & anomaly detection.

Has anyone any idea?does anybody know where can I find them?

Thanks in advance.

[#2]

In-depth video is helpful: https://www.youtube.com/watch?v=5a3HywxY2kQ

https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture#MetronArchitecture-Step1-Tele...

Adding a new event source

https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source

Look through the source:

https://github.com/apache/incubator-metron/tree/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-plat...

Profiling Behavior of Entities

https://github.com/apache/incubator-metron/tree/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-anal...

https://cwiki.apache.org/confluence/display/METRON/Metron+Wiki

Models are stored and data is compared against it.

[#3 · Accepted solution]

There are a variety of meanings of correlation in Metron.

One means is adding correlation keys in the enrichment process, which then allow you to view events together once they are indexed. So you can correlate events by adding a common search key you can then pivot on in Kibana. This is a great means of investigating correlations between alerts and events.

For a more statistical approach to correlation, you will want to look into the profiler which maintains windows of data, which can then be used to correlate time series data using, for example, arima in a model managed by the model as a service infrastructure. This area of Metron is growing quite fast at the moment. I would suggest also looking at the Stellar statistics functions which can be used to build simple anomaly based models as well. It's also easy enough to add functions to Stellar if you want to extend the functionality.

[#4]

@Simon Elliston Ball @Timothy Spann

thank you both for your helpful answers, actually it took me a while to go through your links, but now I know what I needed.

[#5]

Could you elaborate on your findings please ? How can I trigger a complex alarm involving simple alarms from different logs ? Is Stellar of any help for it ?