Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Expert Contributor

Apache Metron - First Steps in the Cloud

Apache Metron integrates a variety of open source technologies to offer a centralized platform for cyber security monitoring, analysis, and threat detection. The platform provides full packet capture indexing, telemetry aggregation and enrichment, advanced behavioral analytics, and the ability to tailor the platform to the specific threats facing your organization. These instruction will take you step-by-step from nothing to a fully-functioning, multi-node Apache Metron cluster running in Amazon's EC2 cloud environment.

The platform will capture live network packets from the wire, leverage best-in-class open source tooling as additional sources of telemetry, and perform real-time enrichment of that data. The data is persisted for advanced analytical modeling and presented within a fully customizable, searchable single pane of glass.

Getting Started

Apache Metron depends on many excellent third-party, open-source components. To see Apache Metron in action, all of these components need to be deployed, configured and connected. To that end, the Apache Metron team has automated as much of the deployment process as possible.

Amazon Web Services

If you already have an Amazon Web Services account that you have used to deploy EC2 hosts then you should be able to skip the next few steps.

1. Head over to Amazon Web Services and create an account. As part of the account creation process you will need to provide a credit card to cover any charges that may apply.

2. Create a set of user credentials through Amazon's Identity and Access Management (IAM) dashboard. On the IAM dashboard menu click "Users" and then "Create New User". Provide a name and ensure that "Generate an access key for each user" remains checked. Download the credentials and keep them for later use.

3200-aws-create-user-credentials.png

3. While still in Amazon's Identity and Access Management (IAM) dashboard, click on the user that was previously created. Click the "Permissions" tab and then the "Attach Policy" button. Attach the following policies to the user.

  • AmazonEC2FullAccess
  • AmazonVPCFullAccess

3221-aws-attach-policy.png

4. Apache Metron uses the official, open source CentOS 6 Amazon Machine Image (AMI). If you have never used this AMI before then you will need to accept Amazon's terms and conditions. Navigate to the web page for this AMI and click the "Continue" button. Choose the "Manual Launch" tab then click the "Accept Software Terms" button.

3222-aws-accept-ami-terms.png

Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us.

Software Prerequisites

The computer used to deploy Apache Metron will need to have Ansible, Python, Maven, SSH, and Git installed. Any platform that supports these tools is suitable, but the following instructions cover only Mac OS X. The easiest means of installing these tools on a Mac is to use the excellent Homebrew project.

1. Install Homebrew by running the following command in a terminal. Refer to the Homebrew home page for the latest installation instructions.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. With Homebrew installed, run the following command in a terminal to install all of the required tools.

brew install ansible brew-pip maven git

3. Ensure that a public SSH key is located at ~/.ssh/id_rsa.pub.

$ cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChv5GJxPjR39UJV7VY17ivbLVlxFrH7UHwh1Jsjem4d1eYiAtde5N2y65/HRNxWbhYli9ED8k0/MRP92ejewucEbrPNq5mytPqdC4IvZ98Ln2GbqTDwvlP3T7xa/wYFOpFsOmXXql8216wSrnrS4f3XK7ze34S6/VmY+lsBYnr3dzyj8sG/mexpJgFS/w83mWJV0e/ryf4Hd7P6DZ5fO+nmTXfKNK22ga4ctcnbZ+toYcPL+ODCh8598XCKVo97XjwF5OxN3vl1p1HHguo3cHB4H1OIaqX5mUt59gFIZcAXUME89PO6NUiZDd3RTstpf125nQVkQAHu2fvW96/f037 nick@localhost

If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered.

ssh-keygen -t rsa

Deploy Apache Metron

1. Download the latest Apache Metron release candidate.

curl https://dist.apache.org/repos/dist/dev/incubator/metron/0.1BETA-RC7-incubating/apache-metron-0.1BETA... | tar xvz
mv incubator-metron-Metron_0.1BETA_rc7 incubator-metron

Build the release candidate.

cd incubator-metron/metron-streaming
mvn package -DskipTests

2. Export the previously generated user credentials in an environment variable. Update the commands below to reflect the access credentials that were previously created for the Apache Metron deployment.

export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ"
export AWS_SECRET_ACCESS_KEY="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"

3. Start the Apache Metron deployment process. The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.

cd ../deployment/amazon-ec2
export EC2_INI_PATH=conf/ec2.ini
ansible-playbook -i ec2.py playbook.yml

Most transient issues can be resolved by simply re-running the playbook. The following errors may occur due to misconfiguration of your Amazon Web Services account. Each of these problems and more are covered in the README document.

Explore Apache Metron

Once the deployment process is complete the terminal will display a set of useful links for exploring Apache Metron.

3223-deploy-complete.png

Multiple hosts have been instantiated to run Apache Metron. To view or manage these hosts login to the Amazon EC2 Dashboard. Each of the hosts has a tag named `env` which is defined as `metron-test` by default. This tag can be used to filter the visible hosts within the Amazon EC2 Dashboard.

3224-ec2-dashboard.png

Each of the provisioned hosts are fully accessible from the internet. Connecting to one over SSH as the user `centos` will not require a password and will authenticate with your SSH key.

ssh centos@ec2-52-91-215-174.compute-1.amazonaws.com

Explore the Apache Metron Dashboard. In this example, the Metron dashboard was installed at http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000, but the specific URL will differ. A direct link for your environment can be found in the terminal output.

3225-metron-dashboard.png

The default Metron deployment is configured to produce enough telemetry data to validate basic functioning of the platform. It is not intended to serve as a production environment. When deployed in this manner, Metron transmits dummy network packet data over a virtual network interface to mimic what might occur in a live production environment.

Each of the default sensors, including Bro, Yaf, and Snort, consume this raw packet data, perform their own unique analyses, and forward on the results. All of this telemetry is then aggregated, enriched, and indexed by Apache Metron. The indexed data is presented within a fully customizable, searchable single pane of glass.

Next Steps

For more information and to get involved, join us at https://metron.incubator.apache.org.

6,316 Views
Comments
avatar
Super Collaborator

I ran into the following error when following these instructions:

2016-04-12 05:42:59,328 p=2472 u=gvetticaden |  fatal: [obfuscated_ip]: UNREACHABLE! => {"changed": false, "msg": "SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue", "unreachable": true}

To fix this issue, see the following thread:

https://community.hortonworks.com/questions/24344/aws-unreachable-error-when-executing-metron-instal...

Version history
Last update:
‎08-17-2019 12:56 PM
Updated by:
Contributors