Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can Apache Metron fullfill my use case

Solved Go to solution

Can Apache Metron fullfill my use case

New Contributor

I have some devices(network switch, router etc), they publish activity logs through syslog. I need to find actionable items from the logs as alerts. An Actionable item could be, a device has been shut down and not restarted, or device disk memory is full etc.

I have following fields in syslogs

timestamp source application_name facility full_message gl2_remote_ip gl2_remote_port gl2_source_input gl2_source_node level message process_id streams

I don't have labelled examples of actionable messages. Can Apache metron do something here with unsupervised learning.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Can Apache Metron fullfill my use case

Contributor

Hi @Sarvesh Kumar

Apache Metron gives you all the tools you need to

  • extract and parse the information from your event. So if the event's message contains the information about if the device has shutdown, you'll be able to create a rule around it.
  • aggregate data and create profiles of devices in certain time windows. So you could create a small function that evaluates the status of a device in a certain time frame and check if the device is up.
  • Disk memory full: If the event source contains the current disk space (and ideally also sends the maximum amount of disk space available) it's just a simple rule to add to create an alert.

Regarding your unsupervised learning question:

  • Your examples don't require machine learning, because they are rule based.
  • You'd want to use machine learning to train a model that generates alerts based on data rather than on rules. (in most cases this is "supervised" learning based on "is alert" or "is not alert").
  • However, Metron provides a "Model as a Service" capabilty, which allows you to deploy models to evaluate events and enrich them.
  • That being said, Metron does not provide models for you. Creating features and models is the data scientists job and depending how thoroughly this is done, this will determine how many accurate alerts (ideally all of them) and how many false positives you have (ideally none).

Hope that helped!

1 REPLY 1
Highlighted

Re: Can Apache Metron fullfill my use case

Contributor

Hi @Sarvesh Kumar

Apache Metron gives you all the tools you need to

  • extract and parse the information from your event. So if the event's message contains the information about if the device has shutdown, you'll be able to create a rule around it.
  • aggregate data and create profiles of devices in certain time windows. So you could create a small function that evaluates the status of a device in a certain time frame and check if the device is up.
  • Disk memory full: If the event source contains the current disk space (and ideally also sends the maximum amount of disk space available) it's just a simple rule to add to create an alert.

Regarding your unsupervised learning question:

  • Your examples don't require machine learning, because they are rule based.
  • You'd want to use machine learning to train a model that generates alerts based on data rather than on rules. (in most cases this is "supervised" learning based on "is alert" or "is not alert").
  • However, Metron provides a "Model as a Service" capabilty, which allows you to deploy models to evaluate events and enrich them.
  • That being said, Metron does not provide models for you. Creating features and models is the data scientists job and depending how thoroughly this is done, this will determine how many accurate alerts (ideally all of them) and how many false positives you have (ideally none).

Hope that helped!