Apache Metron User Personas and Why Metron?

[#1]

Metron User Personas

There are six user personas for Metron:

Persona Name	Description
SOC Analyst	Profile: Beginner, Junior-level analyst Tools Used: SIEM tools/dashboards, Security endpoint UIs, Email/Ticketing/Workflow Systems Responsibilities: Monitor security SIEM tools, search/investigate breaches, malware, review alerts and determine to escalate as tickets or filter out, follow security playbooks, investigate script kiddie attacks.
SOC Investigator	Profile: More advanced SME in cybersecurity, Experienced security analyst, understands more advanced features of security tools, thorough understanding of networking and platform architecture (routers, switches, firewalls, security), Ability to dig through and understand various logs (Network, firewall, proxy, app, etc..) Tools Used: SIEM/Security tools, Scripting languages, SQL, command line Responsibilities: Investigate more complicated/escalated alerts, investigate breaches, Takes the necessary steps to remove/quarantine the malware, breach or infected system, hunter for malware attacks, investigate more complicated attacks like ADT (Advanced Persistent Threats)
SOC Manager	Profile: Experience managing teams, security practitioner that has moved into management. Tools Used: Workflow Systems (e.g: Remedy, JIRA), Ticket/Alerting Systems Responsibilities: Assigns Metron Cases to Analysts. Verifies “completed” metron cases.
Forensic Investigator	Profile: E-discovery experience with security background. Tools Used: SIEM and e-discovery tools Responsibilities: Collect evidence on breach/attack incident, prepare lawyer’s response to breach,
Security Platform Operations Engineer	Profile: Computer Science, developer, and/or Dev/Ops Background. Experience with Big Data technologies and supported distributed applications/systems Tools Used: Security Tools (SIEM, endpoint solutions, UEBA solutions), provisioning, management and monitoring tooling, various programming languages, Big Data and distributing computing platforms. Responsibilities: Helps vet different security tools before bringing them into the enterprise. Establishes best practices and reference architecture with respect to provisioning, management and use of the security tools/ configures the system with respect to deployment/monitoring/etc. Maintains the probes to collect data, enrichment services, loading enrichment data, managing threat feeds, etc..Provides care and feeding of one or more point security solutions. Does capacity planning, system maintenance and upgrades.
Security Data Scientist	Profile: Computer Science / Math Background, security domain experience, dig through as much data as available and looks for patterns and build models Tools Used: Python (scikit learn, Python Notebook), R, Rstudio, SAS, Jupyter, Spark (SparkML) Responsibilities: Work with security data performing data munging, visualization, plotting, exploration, feature engineering and generation, trains, evaluates and scores models

Why Metron? SOC Analyst & Investigator Perspective

The above diagram illustrates the key steps in a typical analyst/investigator workflow. For certain steps in this workflow, Apache Metron provides keys capabilities not found in traditional security tools:

1. Looking through Alerts
   1. Centralized Alerts Console - Having a centralized dashboard for alerts and the telemetry events associated with the alert across all security data sources in your enterprise is a powerful feature within Metron that prevents the Analyst from jumping from one console to another.
   2. Meta Alerts - The long term vision of Metron is to provide a suite of analytical models and packs including Alerts Relevancy Engine and Meta-Alerts. Meta Alerts are generated by groupings or analytics models and provide a mechanism to shield the end user from being inundated with 1000s of granular alerts.
   3. Alerts labeled with threat intel data - Viewing alerts labeled with threat intel from third party feeds allows the analyst to decipher more quickly which alerts are legitimate vs false positives.
2. Collecting Contextual data
   1. Fully enriched messages - Analyst spend a lot of time manually enriching the raw alerts or events. With Metron, analysts work with the fully enriched message.
   2. Single Pane of Glass UI - Single pane of glass that not only has all alerts across different security data sources but also the same view that provides the enriched data
   3. Centralized real-time search - All alerts and telemetry events are indexed in real-time. Hence, the analyst has immediate access to search for all events.
   4. All logs in one place - All events with the enrichments and labels are stored in a single repository.
3. Investigate
   1. Granular access to PCAP - After identifying a legitimate threat, more advanced SOC investigators want the ability to download the raw packet data that caused the alert. Metron provides this capability.
   2. Replay old PCAP against new signatures - Metron can be configured to store raw pcap data in Hadoop for a configurable period of time. This corpus of pcap data can then be replayed to test new analytical models and new signatures.
   3. Tag Behavior for modeling by data scientists
   4. Raw messages used as evidentiary store
   5. Asset inventory and User Identity as enrichment sources.

Note that the above 3 steps in the analyst workflow make up approximately 70% of the time. Metron will drastically decrease the analyst workflow time spend because everything the SOC analyst needs to know is in a single place.

Why Metron? Data Scientist Perspective

The above diagram illustrates the key steps in a typical data science workflow. For certain steps in this workflow, Apache Metron provides key capabilities not found in traditional security tools:

1. Finding the data
   1. All my data is in the same place - One of the biggest challenges faced by security data scientists is to find the data required to train and evaluate the score models. Metron provides a single repository where the enterprise’s security telemetry data are stored.
   2. Data exposed through a variety of APIs - The Metron security vault/repository provides different engines to access and work with the data including SQL, scripting languages, in-memory, java, scala, key-value columnar, REST APIs, User Portals, etc..
   3. Standard Access Control Policies - All data stored in the Metron security vault is secured via Apache Ranger through access policies at a file system level (HDFS) and at processing engine level (Spark, Hive, HBase, Solr, etc..)
2. Cleaning the data
   1. Metron normalizes telemetry events - As discussed in the first blog where we traced an event being processed by the platform, Metron normalizes all telemetry data into at least a standard 7 tuple json structure allowing data scientists to find and correlate data together more easily.
   2. Partial schema validation on ingest - Metron framework will validate data on ingest and will filter out bad data automatically which is something that data scientists, traditionally, spend a lot time doing.
3. Munging Data
   1. Automatic data enrichment - Typically data scientists have to manually enrich data to create and test features or have to work with the data/platform team to do so. With Metron, events are enriched in real-time as it comes in and the enriched event is stored in the Metron security vault.
   2. Automatic application of class labels - Different types of metadata (threat intel information, etc…) is tagged on to the event which allows the data scientists to create feature matrixes for models more easily.
   3. Massively parallel computation framework - All the cleaning and munging of the data is using distributed technologies that allows the processing of these high velocity/ large volumes to be performant and scalable.
4. Visualizing Data
   1. Real-time search + UI - Metron indexes all events and alerts and provides UI dashboard to perform real-time search.
   2. Apache Zeppelin Dashboards - Out of the box Zeppelin dashboards will be available that can be used by SOC analysts. With Zeppelin you can share the dashboards, substitute variables, and can quickly change graph types. An example of a dashboard would be to show all HTTP calls that resulted in 404 errors, visualized as a bar graph ordered by the number of failures.
   3. Integration with Jupyter - Jupyter notebooks will be provided to data scientists for common tasks such as exploration, visualization, plotting, evaluating features, etc..

Note that the above 4 steps in the data science workflow make up approximately 80% of the time. Metron will drastically reduce the time from hypothesis to model for the data scientist.

Apache Metron Core Functional Themes

Now that we have understanding of Metron’s user personas, we will now describe the four core functional themes that Metron will focus on. As the community around Metron continues to group, new features and enhancements will be prioritized across these four themes.

The 4 core functional themes are the following:

Apache Metron Release 0.1 and its Target Personas and Themes

Over the last 4 months, the community led by Hortonworks, has been hard at work on Apache Metron’s first release (Metron 0.1)

Now that we have described the User Personas and core themes for Metron, the following depicts where the engineering focus has been for Metron 0.1.

As the diagram above illustrates, the key focus areas for Metron 0.1 are the following:

• The Platform theme was the primary focus.. Before we can focus on the UI and supporting more telemetry data sources, we need to ensure that the platform is rock hard. This means ensuring an easy way to provision this very complex app and refactor/re-architecture work to ensure code is simpler and easier to maintain, adding new data sources in a declarative manner, performance and extensible improvements and improving the quality of the code.
• The persona of focus is the Security Platform Engineer.
• Metron 0.1 offers dashboard views for the SOC Analyst and SOC investigator.

[#2]

Thanks George, this is a very insightful and deep level of information.