About this article The Metron tutorial article for adding Squid telemetry walks through the process of creating the parser from scratch for Elasticsearch as the Indexing service. This article gives details of extending the tutorial for getting Squid telemetry working with Solr as the backend Indexing service. In other words, these steps are an equivalent of "Installing Squid parser template" for the Elasticsearch. Pre-requisites HCP >= 1.5.0.0 HDP search >= 3.0.0 It is assumed that you have deployed a HCP stack with Solr by following the HCP documentation The Solr node is co-located with the Metron node. In the event that these nodes are on different hosts, ensure that you copy the Metron schema files located at $METRON_HOME/config/schema to the Solr node. It is also assumed that you have followed the Metron tutorial for Squid telemtry by installing the squid sensor, creating the kafka topic and have started the storm topology Steps 1. SSH to the Metron host and run the following commands cd $METRON_HOME/config/schema mkdir squid cd squid Copy the attached files (schema.xml and solrconfig,xml) into the 'squid' folder created above. 2. Run the following commands on the Metron host to create a Solr collection for Squid export SOLR_HOME=/opt/lucidworks-hdpsearch/solr/ export SOLR_USER=solr su $SOLR_USER -c "$SOLR_HOME/bin/solr create -c squid -d $METRON_HOME/config/schema/squid/" 3. Go to the Solr UI at http://<solr-host>:8983/solr/#/~collections to confirm that the Squid collection is present 4. Ingest events into the 'squid' kafka topic and you should see documents being written into the Squid collection in Solr. 5. Fire up Alerts UI and verify that Squid events are seen. ... View more

This article serves as an addendum to the main Metron MaaS README doc in Apache Metron github. It is highly recommended that you go through the README article in github to understand the concepts and working principle. This article only intends to capture the steps specific to the Metron full dev vagrant platform so it is easy for a user to copy-paste-run and get it working quickly. Further, this article only covers the successful startup, deployment and validation of the Metron MaaS service. Refer to the master github README for further steps. Prerequisites * You need to have a working Metron full dev platform before you proceed with the instructions Step 1:Install Required Packages Run the following commands to install Flask, Jinja2, Squid client and the Elasticsearch HEAD plugin: vagrant ssh #To SSH onto the full-dev platform sudo yum install python-flask sudo yum install python-jinja2 sudo yum install squid sudo service start squid sudo /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head Step 2: Create Mock DGA service files Run the following commands: sudo su - metron mkdir mock_dga cd mock_dga Download the files from this link and copy to the folder. Alternativey you use the following commands to create the files: * vi dga.py (paste the below code snippet, save and quit) from flask import Flask from flask import request,jsonify import socket app = Flask(__name__) @app.route("/apply", methods=['GET']) def predict(): h = request.args.get('host') r = {} if h == 'yahoo.com' or h == 'amazon.com': r['is_malicious'] = 'legit' else: r['is_malicious'] = 'malicious' return jsonify(r) if __name__ == "__main__": sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind(('localhost', 0)) port = sock.getsockname()[1] sock.close() with open("endpoint.dat", "w") as text_file: text_file.write("{\"url\" : \"http://0.0.0.0:%d\"}" % port) app.run(threaded=True, host="0.0.0.0", port=port) * vi rest.sh (paste the below code snippet, save and quit) #!/bin/bash python dga.py Run these commands to make the files executable chmod +x /home/metron/mock_dga/* Step 3: Create HDFS directories Run the following commands as vagrant user, and _not_ as metron user sudo su - hdfs -c "hadoop fs -mkdir /user/metron" sudo su - hdfs -c "hadoop fs -chown metron:metron /user/metron"<br> Step 4: Start MaaS service Run the following commands: Note: Change the METRON_HOME variable per the version of Metron you are running sudo su - metron export METRON_HOME=/usr/metron/0.4.2 $METRON_HOME/bin/maas_service.sh -zq node1:2181 Verify MaaS service running and view application log Follow these steps to ensure that the maas service is running properly 1. Launch Ambari UI at http://node1:8080. Authenticate with admin/admin 2. Go to Services -> YARN -> 'Quick Links' dropdown -> ResourceManager UI 3. You should be able to see the application listed in the UI, similar to the below: 4. Click on the application -> Logs -> AppMaster.stderr log file to view the startup logs. Check for presence of any errors. If there are none, you are good to deploy the DGA model in the next step. Step 5: Deploy Mock DGA model Run the following command as metron user to deploy the DGA model $METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -lmp /home/metron/mock_dga -hmp /user/metron/models -mo ADD -m 512 -n dga -v 1.0 -ni 1 Once the command completes, you can monitor the ResourceManager UI application logs to check for any errors. Verify DGA model has been successfully deployed a) Run the following command as metron user: $METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -mo LIST At the end of the command execution, you should be able to see something similar to the following output, which indicates that the model has been successfully deployed. Model dga @ 1.0 dga:1.0 @ http://node1:50451 serving: apply=apply Note: The port number '50451' in the above output may change across different runs. b) Try to hit the model via curl by running the following commands, and verify you are seeing the respective outputs. [metron@node1 ~]$ curl 'http://localhost:50451/apply?host=testing.com' { "is_malicious": "malicious" } [metron@node1 ~]$ curl 'http://localhost:50451/apply?host=yahoo.com' { "is_malicious": "legit" } With this you would have been able to successfully started, deployed and validated Metron MaaS on your full dev Metron platform. Step 6: Squid Example The next steps of sending data through the squid sensor and having it processed through the MaaS is not covered as a part of this article. Please refer to the steps listed in the github README doc. ... View more