Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Super Collaborator

This article serves as an addendum to the main Metron MaaS README doc in Apache Metron github.

It is highly recommended that you go through the README article in github to understand the concepts and working principle. This article only intends to capture the steps specific to the Metron full dev vagrant platform so it is easy for a user to copy-paste-run and get it working quickly.

Further, this article only covers the successful startup, deployment and validation of the Metron MaaS service. Refer to the master github README for further steps.

Prerequisites

* You need to have a working Metron full dev platform before you proceed with the instructions

Step 1:Install Required Packages

Run the following commands to install Flask, Jinja2, Squid client and the Elasticsearch HEAD plugin:

vagrant ssh #To SSH onto the full-dev platform
sudo yum install python-flask
sudo yum install python-jinja2
sudo yum install squid
sudo service start squid
sudo /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head

Step 2: Create Mock DGA service files

Run the following commands:

sudo su - metron
mkdir mock_dga
cd mock_dga

Download the files from this link and copy to the folder. Alternativey you use the following commands to create the files:

* vi dga.py

(paste the below code snippet, save and quit)

from flask import Flask
from flask import request,jsonify
import socket
app = Flask(__name__)
@app.route("/apply", methods=['GET'])
def predict():
        h = request.args.get('host')
        r = {}
        if h == 'yahoo.com' or h == 'amazon.com':
                r['is_malicious'] = 'legit'
        else:
                r['is_malicious'] = 'malicious'
        return jsonify(r)
if __name__ == "__main__":
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.bind(('localhost', 0))
        port = sock.getsockname()[1]
        sock.close()
        with open("endpoint.dat", "w") as text_file:
                text_file.write("{\"url\" : \"http://0.0.0.0:%d\"}" % port)
        app.run(threaded=True, host="0.0.0.0", port=port)

* vi rest.sh

(paste the below code snippet, save and quit)

#!/bin/bash
python dga.py

Run these commands to make the files executable

chmod +x /home/metron/mock_dga/*

Step 3: Create HDFS directories

Run the following commands as vagrant user, and _not_ as metron user

sudo su - hdfs -c "hadoop fs -mkdir /user/metron"
sudo su - hdfs -c "hadoop fs -chown metron:metron /user/metron"<br>

Step 4: Start MaaS service

Run the following commands:

Note: Change the METRON_HOME variable per the version of Metron you are running

sudo su - metron
export METRON_HOME=/usr/metron/0.4.2
$METRON_HOME/bin/maas_service.sh -zq node1:2181

Verify MaaS service running and view application log

Follow these steps to ensure that the maas service is running properly

1. Launch Ambari UI at http://node1:8080. Authenticate with admin/admin

2. Go to Services -> YARN -> 'Quick Links' dropdown -> ResourceManager UI

3. You should be able to see the application listed in the UI, similar to the below:

43729-maas-resourcemanagerui.png

4. Click on the application -> Logs -> AppMaster.stderr log file to view the startup logs. Check for presence of any errors. If there are none, you are good to deploy the DGA model in the next step.

Step 5: Deploy Mock DGA model

Run the following command as metron user to deploy the DGA model

$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -lmp /home/metron/mock_dga -hmp /user/metron/models -mo ADD -m 512 -n dga -v 1.0 -ni 1

Once the command completes, you can monitor the ResourceManager UI application logs to check for any errors.

Verify DGA model has been successfully deployed

a) Run the following command as metron user:

$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -mo LIST

At the end of the command execution, you should be able to see something similar to the following output, which indicates that the model has been successfully deployed.

Model dga @ 1.0
dga:1.0 @ http://node1:50451 serving:
 apply=apply

Note: The port number '50451' in the above output may change across different runs.

b) Try to hit the model via curl by running the following commands, and verify you are seeing the respective outputs.

[metron@node1 ~]$ curl 'http://localhost:50451/apply?host=testing.com'
{
  "is_malicious": "malicious"
}

[metron@node1 ~]$ curl 'http://localhost:50451/apply?host=yahoo.com'
{
  "is_malicious": "legit"
}

With this you would have been able to successfully started, deployed and validated Metron MaaS on your full dev Metron platform.

Step 6: Squid Example

The next steps of sending data through the squid sensor and having it processed through the MaaS is not covered as a part of this article. Please refer to the steps listed in the github README doc.

1,120 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 09:59 AM
Updated by:
 
Contributors
Top Kudoed Authors