Created on 04-05-201609:23 PM - edited 08-17-201912:56 PM
Apache Metron - First Steps in the Cloud
Apache Metron integrates a variety of open source technologies to offer a centralized platform for cyber security monitoring, analysis, and threat detection. The platform provides full packet capture indexing, telemetry aggregation and enrichment, advanced behavioral analytics, and the ability to tailor the platform to the specific threats facing your organization.
These instruction will take you step-by-step from nothing to a fully-functioning, multi-node Apache Metron cluster running in Amazon's EC2 cloud environment.
The platform will capture live network packets from the wire, leverage best-in-class open source tooling as additional sources of telemetry, and perform real-time enrichment of that data. The data is persisted for advanced analytical modeling and presented within a fully customizable, searchable single pane of glass.
Apache Metron depends on many excellent third-party, open-source components. To see Apache Metron in action, all of these components need to be deployed, configured and connected. To that end, the Apache Metron team has automated as much of the deployment process as possible.
Amazon Web Services
If you already have an Amazon Web Services account that you have used to deploy EC2 hosts then you should be able to skip the next few steps.
1. Head over to Amazon Web Services and create an account. As part of the account creation process you will need to provide a credit card to cover any charges that may apply.
2. Create a set of user credentials through Amazon's Identity and Access Management (IAM) dashboard. On the IAM dashboard menu click "Users" and then "Create New User". Provide a name and ensure that "Generate an access key for each user" remains checked. Download the credentials and keep them for later use.
3. While still in Amazon's Identity and Access Management (IAM) dashboard, click on the user that was previously created. Click the "Permissions" tab and then the "Attach Policy" button. Attach the following policies to the user.
Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us.
The computer used to deploy Apache Metron will need to have Ansible, Python, Maven, SSH, and Git installed. Any platform that supports these tools is suitable, but the following instructions cover only Mac OS X. The easiest means of installing these tools on a Mac is to use the excellent Homebrew project.
1. Install Homebrew by running the following command in a terminal. Refer to the Homebrew home page for the latest installation instructions.
If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered.
ssh-keygen -t rsa
Deploy Apache Metron
1. Download the latest Apache Metron release candidate.
mvn package -DskipTests
2. Export the previously generated user credentials in an environment variable. Update the commands below to reflect the access credentials that were previously created for the Apache Metron deployment.
3. Start the Apache Metron deployment process. The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.
ansible-playbook -i ec2.py playbook.yml
Most transient issues can be resolved by simply re-running the playbook. The following errors may occur due to misconfiguration of your Amazon Web Services account. Each of these problems and more are covered in the README document.
Once the deployment process is complete the terminal will display a set of useful links for exploring Apache Metron.
Multiple hosts have been instantiated to run Apache Metron. To view or manage these hosts login to the Amazon EC2 Dashboard. Each of the hosts has a tag named `env` which is defined as `metron-test` by default. This tag can be used to filter the visible hosts within the Amazon EC2 Dashboard.
Each of the provisioned hosts are fully accessible from the internet. Connecting to one over SSH as the user `centos` will not require a password and will authenticate with your SSH key.
The default Metron deployment is configured to produce enough telemetry data to validate basic functioning of the platform. It is not intended to serve as a production environment. When deployed in this manner, Metron transmits dummy network packet data over a virtual network interface to mimic what might occur in a live production environment.
Each of the default sensors, including Bro, Yaf, and Snort, consume this raw packet data, perform their own unique analyses, and forward on the results. All of this telemetry is then aggregated, enriched, and indexed by Apache Metron. The indexed data is presented within a fully customizable, searchable single pane of glass.