This article serves as an addendum to the main Metron MaaS
README doc in Apache Metron github. It is highly recommended that you go through the README article in github to understand the concepts and working principle. This article only intends to capture the steps specific to the Metron full dev vagrant platform so it is easy for a user to copy-paste-run and get it working quickly. Further, this article only covers the successful startup, deployment and validation of the Metron MaaS service. Refer to the master github README for further steps. Prerequisites
* You need to have a working Metron full dev platform before you proceed with the instructions Step 1:Install Required Packages
Run the following commands to install Flask, Jinja2, Squid client and the Elasticsearch HEAD plugin:
vagrant ssh #To SSH onto the full-dev platform
sudo yum install python-flask
sudo yum install python-jinja2
sudo yum install squid
sudo service start squid
sudo /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
Step 2: Create Mock DGA service files
Run the following commands:
sudo su - metron
mkdir mock_dga
cd mock_dga
Download the files from this
link and copy to the folder. Alternativey you use the following commands to create the files:
* vi dga.py
(paste the below code snippet, save and quit)
from flask import Flask
from flask import request,jsonify
import socket
app = Flask(__name__)
@app.route("/apply", methods=['GET'])
def predict():
h = request.args.get('host')
r = {}
if h == 'yahoo.com' or h == 'amazon.com':
r['is_malicious'] = 'legit'
else:
r['is_malicious'] = 'malicious'
return jsonify(r)
if __name__ == "__main__":
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('localhost', 0))
port = sock.getsockname()[1]
sock.close()
with open("endpoint.dat", "w") as text_file:
text_file.write("{\"url\" : \"http://0.0.0.0:%d\"}" % port)
app.run(threaded=True, host="0.0.0.0", port=port)
* vi rest.sh
(paste the below code snippet, save and quit)
#!/bin/bash
python dga.py
Run these commands to make the files executable
chmod +x /home/metron/mock_dga/*
Step 3: Create HDFS directories
Run the following commands as
vagrant user, and _not_ as metron user
sudo su - hdfs -c "hadoop fs -mkdir /user/metron"
sudo su - hdfs -c "hadoop fs -chown metron:metron /user/metron"<br>
Step 4: Start MaaS service
Run the following commands:
Note: Change the METRON_HOME variable per the version of Metron you are running
sudo su - metron
export METRON_HOME=/usr/metron/0.4.2
$METRON_HOME/bin/maas_service.sh -zq node1:2181
Verify MaaS service running and view application log
Follow these steps to ensure that the maas service is running properly
1. Launch Ambari UI at http://node1:8080. Authenticate with admin/admin
2. Go to Services -> YARN -> 'Quick Links' dropdown -> ResourceManager UI
3. You should be able to see the application listed in the UI, similar to the below:
4. Click on the application -> Logs -> AppMaster.stderr log file to view the startup logs. Check for presence of any errors. If there are none, you are good to deploy the DGA model in the next step. Step 5: Deploy Mock DGA model
Run the following command as metron user to deploy the DGA model
$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -lmp /home/metron/mock_dga -hmp /user/metron/models -mo ADD -m 512 -n dga -v 1.0 -ni 1
Once the command completes, you can monitor the ResourceManager UI application logs to check for any errors. Verify DGA model has been successfully deployed
a) Run the following command as metron user:
$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -mo LIST
At the end of the command execution, you should be able to see something similar to the following output, which indicates that the model has been successfully deployed. Model dga @ 1.0
dga:1.0 @ http://node1:50451 serving:
apply=apply
Note: The port number '50451' in the above output may change across different runs.
b) Try to hit the model via curl by running the following commands, and verify you are seeing the respective outputs. [metron@node1 ~]$ curl 'http://localhost:50451/apply?host=testing.com'
{
"is_malicious": "malicious"
}
[metron@node1 ~]$ curl 'http://localhost:50451/apply?host=yahoo.com'
{
"is_malicious": "legit"
}
With this you would have been able to successfully started, deployed and validated Metron MaaS on your full dev Metron platform. Step 6: Squid Example The next steps of sending data through the squid sensor and having it processed through the MaaS is not covered as a part of this article. Please refer to the steps listed in the github README doc.
... View more
