Created on 04-26-2017 12:26 PM - edited 08-17-2019 09:04 AM
For Generating RPMs and Management Pack
For Metron Cluster deployment
In this step, you will create the RPMs and Ambari management pack tarball on your local system. The RPMs and Ambari mpack need to be SCP'ed into the cluster hosts
a) Build Mpack
On your local system where you have cloned the Git repo, run the following command:
cd incubator-metron mvn clean package -DskipTests
Once the above command has run successfully, the metron management pack will be generated in the path at:
b) Build RPMs
Run the following commands to generate the RPMs:
Note: Docker service should be running in order for the below command to work properly.
cd incubator-metron/metron-deployment/ mvn clean package -Pbuild-rpms -DskipTests
The above command will build the RPMs and create them under
c) Copy RPMs and Mpack to cluster nodes
Use either SCP or your favorite file transfer application to copy the mpack and RPMs over the to the cluster
metron_mpack-xxxx.tar.gzfile to the node which is going to run Ambari Server (For E.g. node #1)
scpall of the generated metron rpms to the
SSH to the node where Ambari server needs to be installed (node #1) and follow these steps.
Ensure CentOS repo for Ambari server is updated:
wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/18.104.22.168/ambari.repo -O /etc/yum.repos.d/ambari.repo
Install Ambari server, install mpack and start service
yum install ambari-server -y ambari-server setup -s ambari-server install-mpack --mpack=/path/to/metron_mpack-22.214.171.124-SNAPSHOT.tar.gz --verbose ambari-server start
Point your browser to
http://<node#1>:8080 to start the cluster installation wizard. Follow the screenshots below to complete the Metron installation.
Start the wizard by choosing the "Create a Cluster" option and specify a name for the cluster.
Select HDP 2.5 stack for Metron 0.3.x from the version selection page.
Specify the list of hosts in your cluster along with the connection information (e.g. private key)
In the "Choose Services" page, select the following services -- which are required as a minimum for a working Metron deployment. You may select other additional services as per your need.
Here are sample screenshots:
In the "Assign Masters" page, ensure the following criteria is met. Observe that the Ambari wizard will display warning / error popups when the below criteria is not met.
a) Ensure that the Kibana Server, Metron Enrichment, Metron Indexing and Metron Parsers components are all assigned to the _same node_. It is important to note that these _should not_ be on the same node as Ambari server. All of these, for e.g. could reside on the node #12
b) It is preferred to run the Elasticsearch Master on the same node as Metron components (node #12, in this example).
c) Add up to 4 Kafka Broker components. Ensure that one of the Kafka Broker components is installed on the Metron node #12.
d) [Optional] You may retain the Zookeeper server on only one host and remove the rest
The other components may be left at their defaults
Note: For some of the components, there is a warning message to remind the user for client selection. Choose "Continue Anyway" if the Validation Issues warning dialog pops up.
In the "Assign Slaves and Clients" window, you need to ensure that the Metron node (#12 in this example) is selected for the following:
Optionally, you can choose to install "Client" on all the nodes.
In the "Customize Services" window, refer to the screenshots below for filling in the respective tabs
1. Change NameNode heap size from the default 1024 MB to about 4096 MB
2. Under Elasticsearch -> Advanced elastic-site -> zen_discovery_ping_unicast_hosts, specify the location where Elasticsearch master is installed (E.g. node #12 in this case.
3. Under Kibana -> Advanced kibana-env, specify the kibana_es_url to the Elasticsearch master node URL with port as 9200.
4. Under Metron, change the parameters in the respective tabs below.
Tab #1 - Default Settings for Metron services.
Provide details of the Elasticsearch hosts.
Tab #2 - Repository settings - Remote vs. Local repository
If you choose to install using a Local repository, ensure that you have copied the Metron RPMs into the /localrepo folder in the Metron node.
If you choose to install using a remote repository, specify the URL where the repo file is available.
Hit the "Next" button and then hit the "Deploy" button to proceed with Metron deployment. This will start the cluster deployment and in time, all the services should be up and running.