Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Rising Star

Metron TP1 Features

The following are key capabilities available in Metron TP1 broken up across its four key functional themes.

3527-metron-tp1-features-4.png

How do I get Started?

You can spin up the Metron TP1 in two ways:

  • Fully Automated 10 Node Ansible Based Install on AWS using Ambari Blueprints and AWS APIs
    • If you want a more realistic setup of the Metron app, use this approach. Keep in mind that this install will spin up 10 m4.xlarge EC2 instance by default
    • Detailed instructions how to do the install can be found in the following HCC Article: Apache Metron - First Steps in the Cloud

Where do I get Help?

Hortonworks has created new Track called CyberSecurity in the Hortonworks Community Connection (HCC). The link to the this new track in HCC is the following: HCC CyberSecurity Track.

Apache Metron committers are subscribed to this track and are constantly monitoring it for any questions the community has on TP1.

When asking a question about Metron TP1, please select the “CyberSecurity” Track and add the following tags: “Metron” and “tech-preview”.

Platform Theme Features of Metron TP1

The below is a summary of the key platform features added in TP1:

FeatureRelated Apache Metron JIRAS
Support for HDP 2.3
Refactor Metron Topologies for Performance, Easier Manageability & Supportability

METRON-56 METRON-33

Fully Automated Install of Metron on AWS on multi-node HDP cluster via Ansible scripts, Ambari blueprints and APIs.

METRON-59 METRON-77 METRON-76 METRON-69 METRON-63 METRON-61 METRON-43 METRON-2

Single Node Vagrant Support for Metron for Development

METRON-21

Unit and Integration Testing Frameworks, Code Test CoverageMETRON-82 METRON-58 METRON-37 METRON-28

Telemetry Data Source Theme Features of Metron TP1

Metron TP1 focus is network telemetry data sources as described below. They represent the most valuable granular data one can collect and perform next generation analytics on.

The Key Data collection features for Metron TP1 are the following:

FeatureRelated Apache Metron JIRAS
PCAP Ingest Data Services - Performant C++ probe that captures network packet and streams them into Kafka and gets bulk loaded into MetronMETRON-79 METRON-79 METRON-73 METRON-55 METRON-39
YAF/Netflow Ingest Data Services - Ingests netflow data into MetronMETRON-67 METRON-60
Bro Ingest Data Services - Custom BRO plugin that pushes out DPI (Deep Packet Inspection) metadata into MetronMETRON-25 METRON-73 METRON-64
Snort Ingest Data Services - Stream snort generated alerts via Flume into MetronMETRON-57
Grok Framework - Ability to add new Data Sources to Metron without writing new Parsing Topologies. For each new data source, grok expression file can be provided to normalized into Metron Event. METRON-66

Real-time Data Processing Theme Features of Metron TP1

For this theme, the key features in Metron TP1 are the following:

FeatureRelated Apache Metron JIRAS
Enrichment Services - OOO support for GeoIP and Host enrichments, extensible framework to plug-in new enrichments, & management Utilities for Enrichment Data METRON-32 METRON-43
Threat Intel Services - Integration with Soltra (Threat Intel Aggregrator) and Hail a Taxii, management Utilities for Threat Intel (Streaming and Bulk Load, aging out of data)

METRON-35 METRON-50

Alerting Services - Alerts can be fired via a snort event or intel threat feed hit

Indexing Services - Support for indexing via ElasticSearch

METRON-36 METRON-56 METRON-66

Storage Services - persisting all enrichment telemetry data in HDFS and or HBaseMETRON-62 METRON-22

UI Theme Features of Metron TP1

There was less focus on the UI Theme but Metron TP1 does provide the following new UI features:

FeatureRelated Apache Metron JIRAS
Metron Investigator IO Dashboard for the SOC Analyst and Investigator Personas built on top of KibanaMETRON-72 METRON-77 METRON-81
Histogram Panels for each of the data sources (YAF, Bro, Snort, PCAP)METRON-60 METRON-52
PCAP panel allow you to search for and download PCAP filesMETRON-72 METRON-77 METRON-81
Ability to customize the Metron UI with different data sources and different panel types.METRON-72 METRON-77 METRON-81

2,471 Views
Comments
Not applicable

Is it possible to run this on Solr versus elastic? Solr is more closely tied to the Lucene project, uses the same Zookeeper based distributed coordination and continues to add more features to the open source project (graph query, parallel sql), while elastic seems to be taking more new features closed source. I've been building a lot of related features for network analysis in Solr.

Rising Star

Good question @Matt McKnight. We will have support for Solr indexing services in Metron TP2 which is slated for end of May. However in TP2, we will still only support Metron UI that is based on Kibana (based on Elastic). This will change in subsequent reelases. So net net, by middle/end of May we will support Solr indexing but you would have to write the UI that calls the SOLR Apis for search queries. Farther down the line, we will provide a custom UI (away from Kibana) that uses SOLR to do search.

Make sense?

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 12:54 PM
Updated by:
 
Contributors
Top Kudoed Authors