Hello from the Metron PM and Eng Team Today, the Hortonworks Metron product management and engineering team are kicking off a multi-part blog series on Apache Metron, the next gen security analytics application that Hortonworks is building working with the Apache Community. Over the course of the next few weeks, we will release a series of blogs that covers the following topics: Part 1 - Apache Metron Explained - Overview of Apache Metron and traces a security telemetry event as it flows through the platform. Part 2 - Apache Metron User Personas and Why Metron? - Who will be the different users of Apache Metron? What are the core functional themes? What has been the focus for the first release? We will address all 3 of these questions in this blog. Part 3 - Apache Metron Tech Preview 1 - Come and Get It We will walk you through what the Metron community has been working on for the last 3 months. By the end of this blog, you will have a good understanding of what is in Metron Tech Preview 1 and how to get it installed, deployed and building on top of it. Part 4: Apache Metron UI and Finding a Needle in the Haystack Use Case - We will walkthrough the Metron UI components and how SOC Analyst would use it for common Metron Use Cases. Part 5 - Deep Dive on Apache Metron Tech Preview 1 - We will double click on the major functional areas of Metron TP 1. Part 6 - Apache Metron Vision - With a solid understand of what TP1 consists of, this blog will provide a glimpse into the roadmap and vision for Apache Metron and what the project will look like by the end of 2016 focusing on the analytics work planned. Roots of Apache Metron To understand Apache Metron, we have to first start with the origins of the project which emerged from the Cisco Project called OpenSoc. The below diagram highlights some of the key events in the history of Apache Metron starting with Cisco OpenSoc. 2005 to 2008 The Problem - Cyber crime spiked significantly and a severe shortage of security talent arose. The first set of companies alerted to this issue are high profile banks and large organizations with interesting proprietary information to state sponsored agents. All of the best investigators and analysts were gobbled up by multinational banking and financial services firms, large hospitals, telcos, and defense contractors. The Rise of a New Industry, the Managed SOC - Those who could not acquire security talent were still in need of a team. Cisco was sitting on a gold mine of security talent that they had accumulated over the years. Utilizing this talent, they p­roduced a managed service offering around managed security operations centers. Post 2008 The Age of Big Data Changed Everything - The Age of Big Data arrived, bringing more streaming data, virtualized infrastructure, data centers emitting machine exhaust from VMs, and Bring Your Own Device programs. The amount of data exploded and so did the cost of the required tools like traditional SIEMs. These tools became cost prohibitive as they changed to data driven licensing structures. Cisco’s ability to operate the managed SOC with these tools was in jeopardy and security appliance vendors took control of the market. 2013 OpenSOC is Born and Hadoop Matures - Cisco decided to build a toolset of their own. They didn’t just want to replace these tools but they wanted to improve and modernize them, taking advantage of open source. Cisco released its managed SOC service to the community as Hadoop matured and Storm became available. It was a perfect combination of a use case need and technology. OpenSOC was the first project to take advantage of Storm, Hadoop, and Kafka, as well as migrate the legacy ways into a forward thinking future type paradigm. September 2013 thru April 2015 The Origins of Apache Metron - For about 24 months, a Cisco team, led by their chief data scientist James Sirota, with the help of a Hortonworks team, led by platform architect Sheetal Dolas, worked to create a next generation managed SOC service built on top of open source big data technologies. The Cisco OpenSOC managed SOC offering went into production for a number of customers in April of 2015. A short time after, Cisco made a couple of acquisitions that brought in third party technologies transforming OpenSOC into a closed source, hardware based version. October 2015 OpenSOC Chief Data Scientist Joins Hortonworks - James Sirota, the chief data scientist and lead of the Cisco OpenSOC initiative, leaves Cisco to join Hortonworks. Over the course of the next 4 months, James starts to build a rock star engineering team at Hortonworks with the focus of building an open-source CyberSecurity application. December 2015 Metron Accepted into Apache Incubation - Hortonworks, with the help and support of key Apache community partners, including ManTech, B23 and others, submit Metron (renamed from OpenSOC) as an Apache incubator project. In December of 2015, the project is accepted into Apache incubation. Hortonworks and the community innovate at impressive speeds to add new features to Apache Metron and harden the platform. The Metron team builds an extensible, open architecture to account for the variety of tools used in customer environments (thousands of firewalls, thousands of domains and a multitude of Intrusion Detection Systems). Metron’s open approach makes it much easier to tailor to the community’s use cases. April 2016 First official Release of Apache Metron 0.1 - After 4 months of hard work and rapid innovation by the Metron community, Apache Metron’s first release Metron 0.1 is cut. Given Hortonworks proven commitment to the Apache Software Foundation process and our track record for creating and leading robust communities, we feel uniquely qualified to bring this important technology and its capabilities to the broader open source community. Without Hortonworks, the Apache Metron project would not exist today! ... View more

Introduction If you are new to Metron or the Metron Tech Preview 1, the following links should provide some good information to review before walking through the installation: Intro to Apache Metron What is in Apache Metron Tech Preview 1 Build Instructions The following steps provide instructions on how to install a full working Metron application on a single node VM with Vagrant. This deployment option is ideal for experimenting and playing with the Metron application. While these instructions should work on most development environments, these instructions were tested on Mac OS X El Capitan. Prerequisites On your Macintosh: Install the latest version of Virtual Box. Install the latest version of Vagrant. Install Maven if you don't have it, and define associated environmental variables. For example, add the following to your ~/.bash_profile file: export MAVEN_HOME=/Users/rmckissick/Documents/Files/apache-maven-3.3.9 export PATH=$MAVEN_HOME/bin:$PATH Install JAVA 1.8 if you don't have it, and define associated environment variables. For example, add the following your ~/.bash_profile file. export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home export PATH=$JAVA_HOME/bin:$PATH If you installed Maven and Java and edited your profile file in steps 2 and 3, reload .bash_profile: source~/.bash_profile Check your Maven installation: mvn–version You should see information about Maven, Java, and OS X. Install Ansible, version 2.0 or greater. For example: sudo su - easy_install pip export CFLAGS=-Qunused-arguments export CPPFLAGS=-Qunused-arguments pip install ansible exit (exit logs off from root and returns to your user account) Build Apache Metron Download the 0.1 Metron binaries from here (download the .tar.gz file). Untar the binaries to a location that will be easy to find later: tar -zxvf apache-metron-0.1BETA-RC7-incubating.tar.gz Build the Metron application: cd incubator-metron-Metron_0.1BETA_rc7 mvn apache-rat:check && cd metron-streaming && mvn clean integration-test && cd .. The mvn command downloads and builds Metron components. It should take about 15 minutes, depending on your hardware configuration. When it finishes, you should see a message similar to the following: [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Metron-Streaming ................................... SUCCESS [ 31.437 s] [INFO] Metron-Common ...................................... SUCCESS [04:58 min] [INFO] Metron-EnrichmentAdapters .......................... SUCCESS [ 14.185 s] [INFO] Metron-MessageParsers .............................. SUCCESS [ 2.704 s] [INFO] Metron-Indexing .................................... SUCCESS [ 26.989 s] [INFO] Metron-Alerts ...................................... SUCCESS [ 4.651 s] [INFO] Metron-Testing ..................................... SUCCESS [ 9.167 s] [INFO] Metron-DataLoads ................................... SUCCESS [04:26 min] [INFO] Metron-Topologies .................................. SUCCESS [03:05 min] [INFO] Metron-Pcap_Service ................................ SUCCESS [ 43.666 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14:43 min [INFO] Finished at: 2016-04-26T13:11:09-07:00 [INFO] Final Memory: 122M/1649M Deploy Metron as a single VM via Vagrant and Ansible: cd deployment/vagrant/singlenode-vagrant vagrant plugin install vagrant-hostmanager vagrant up The vagrant up process will run through a series of Ansible scripts, installing Ambari, HDP, and Metron on the single-node VM. The process should take about 45 - 60 minutes depending on your hardware configuration. Verify That Apache Metron is Deployed Successfully Check Ambari to make sure all the services are up by going to Ambari. Sign on with the default login and password "admin". The Ambari dashboard should look like the following: Verify that four Storm topologies have been deployed: bro, enrichment, snort, and yaf. From Ambari, navigate to Storm -> Quick Links -> Storm UI. You should see the four storm topologies deployed. The Metron Storm UI should look something like the following: Check that the enrichment topology has emitted some data (this could take a few minutes to show up in the Storm UI). The storm enrichment topology UI should look something like the following: Go to the Metron UI (at http://node1:5000). Check indexes to make sure indexing is done correctly and data is visualized. The Metron UI should look something like the following: Check that some data is written into HDFS for at least one of the data sources vagrant ssh node1 sudo su hdfs hadoop fs -ls /apps/metron/enrichment/indexed Questions/Issues If you have any questions or install issues, post your question to the CyberSecurity HCC Track. ... View more