Can Apache Metron fullfill my use case

[#1]

I have some devices(network switch, router etc), they publish activity logs through syslog. I need to find actionable items from the logs as alerts. An Actionable item could be, a device has been shut down and not restarted, or device disk memory is full etc.

I have following fields in syslogs

timestamp source application_name facility full_message gl2_remote_ip gl2_remote_port gl2_source_input gl2_source_node level message process_id streams

I don't have labelled examples of actionable messages. Can Apache metron do something here with unsupervised learning.

[#2 · Accepted solution]

Hi @Sarvesh Kumar

Apache Metron gives you all the tools you need to

• extract and parse the information from your event. So if the event's message contains the information about if the device has shutdown, you'll be able to create a rule around it.
• aggregate data and create profiles of devices in certain time windows. So you could create a small function that evaluates the status of a device in a certain time frame and check if the device is up.
• Disk memory full: If the event source contains the current disk space (and ideally also sends the maximum amount of disk space available) it's just a simple rule to add to create an alert.

Regarding your unsupervised learning question:

• Your examples don't require machine learning, because they are rule based.
• You'd want to use machine learning to train a model that generates alerts based on data rather than on rules. (in most cases this is "supervised" learning based on "is alert" or "is not alert").
• However, Metron provides a "Model as a Service" capabilty, which allows you to deploy models to evaluate events and enrich them.
• That being said, Metron does not provide models for you. Creating features and models is the data scientists job and depending how thoroughly this is done, this will determine how many accurate alerts (ideally all of them) and how many false positives you have (ideally none).

Hope that helped!