Note:
- This article is an extension of the instructions at this link.
- This article uses the example of a 12-node CentOS 7 VM cluster on Openstack environment and with HDP 2.5 as base stack for Metron ver > 0.3.x (or HDP 2.4 for Metron < 0.2.x).
- Please be aware that Metron deployment using Ambari management packs is actively being enhanced and worked upon. Some of the steps in this article and/or the behavior might be altered or become obsolete. Refer to this link for the current set of limitations with the Ambari Mpack installation and setup.
- The selection of nodes, services, slaves and clients in the Ambari cluster wizard are an indicative example. Optimization for performance and scale requirements are out of scope of this article.
Prerequisites
For Generating RPMs and Management Pack
- Local system (Mac or Linux) with build tools installed viz. Maven and docker. Refer here for more details on the tool pre-requisites.
- Docker service has been installed and is running.
- A cloned copy of the Metron git repo.
For Metron Cluster deployment
- 12 VMs running CentOS 7.x.
- JDK > 1.8.x installed on all nodes.
- Ambari Server > 2.4.2.x required
Step 1 - Build Metron RPMs and Management Pack
In this step, you will create the RPMs and Ambari management pack tarball on your local system. The RPMs and Ambari mpack need to be SCP'ed into the cluster hosts
a) Build Mpack
On your local system where you have cloned the Git repo, run the following command:
cd incubator-metron mvn clean package -DskipTests
Once the above command has run successfully, the metron management pack will be generated in the path at:
incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-xxxx.tar.gz
b) Build RPMs
Run the following commands to generate the RPMs:
Note: Docker service should be running in order for the below command to work properly.
cd incubator-metron/metron-deployment/ mvn clean package -Pbuild-rpms -DskipTests
The above command will build the RPMs and create them under incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch
c) Copy RPMs and Mpack to cluster nodes
Use either SCP or your favorite file transfer application to copy the mpack and RPMs over the to the cluster
- Copy the
metron_mpack-xxxx.tar.gzfile to the node which is going to run Ambari Server (For E.g. node #1) - On the node where you would like to install Metron (For e.g. node 12), create a directory call
/localrepoandscpall of the generated metron rpms to the/localrepofolder
Step 2 - Install Ambari Server and Metron management pack
SSH to the node where Ambari server needs to be installed (node #1) and follow these steps.
Ensure CentOS repo for Ambari server is updated:
wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
Install Ambari server, install mpack and start service
yum install ambari-server -y ambari-server setup -s ambari-server install-mpack --mpack=/path/to/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose ambari-server start
Step 3 - Cluster Installation
Point your browser to http://<node#1>:8080 to start the cluster installation wizard. Follow the screenshots below to complete the Metron installation.
Start the wizard by choosing the "Create a Cluster" option and specify a name for the cluster.
Select HDP 2.5 stack for Metron 0.3.x from the version selection page.
Specify the list of hosts in your cluster along with the connection information (e.g. private key)
In the "Choose Services" page, select the following services -- which are required as a minimum for a working Metron deployment. You may select other additional services as per your need.
- HDFS
- YARN + MapReduce2
- HBase
- HIVE
- Pig
- Zookeeper
- Storm
- Spark
- Kafka
- Zeppelin
- Elasticsearch
- Kibana
- Metron
- Slider
Here are sample screenshots:
In the "Assign Masters" page, ensure the following criteria is met. Observe that the Ambari wizard will display warning / error popups when the below criteria is not met.
a) Ensure that the Kibana Server, Metron Enrichment, Metron Indexing and Metron Parsers components are all assigned to the _same node_. It is important to note that these _should not_ be on the same node as Ambari server. All of these, for e.g. could reside on the node #12
b) It is preferred to run the Elasticsearch Master on the same node as Metron components (node #12, in this example).
c) Add up to 4 Kafka Broker components. Ensure that one of the Kafka Broker components is installed on the Metron node #12.
d) [Optional] You may retain the Zookeeper server on only one host and remove the rest
The other components may be left at their defaults
Note: For some of the components, there is a warning message to remind the user for client selection. Choose "Continue Anyway" if the Validation Issues warning dialog pops up.
In the "Assign Slaves and Clients" window, you need to ensure that the Metron node (#12 in this example) is selected for the following:
- DataNode
- NodeManager
- RegionServer
- Supervisor
- Flume
- ElasticsearchDataNode
- Client
Optionally, you can choose to install "Client" on all the nodes.
In the "Customize Services" window, refer to the screenshots below for filling in the respective tabs
1. Change NameNode heap size from the default 1024 MB to about 4096 MB
2. Under Elasticsearch -> Advanced elastic-site -> zen_discovery_ping_unicast_hosts, specify the location where Elasticsearch master is installed (E.g. node #12 in this case.
3. Under Kibana -> Advanced kibana-env, specify the kibana_es_url to the Elasticsearch master node URL with port as 9200.
4. Under Metron, change the parameters in the respective tabs below.
Tab #1 - Default Settings for Metron services.
Provide details of the Elasticsearch hosts.
Tab #2 - Repository settings - Remote vs. Local repository
If you choose to install using a Local repository, ensure that you have copied the Metron RPMs into the /localrepo folder in the Metron node.
If you choose to install using a remote repository, specify the URL where the repo file is available.
Hit the "Next" button and then hit the "Deploy" button to proceed with Metron deployment. This will start the cluster deployment and in time, all the services should be up and running.
