Created on 04-26-201712:26 PM - edited 08-17-201909:04 AM
This article is an extension of the instructions at this link.
This article uses the example of a 12-node CentOS 7 VM cluster on Openstack environment and with HDP 2.5 as base stack for Metron ver > 0.3.x (or HDP 2.4 for Metron < 0.2.x).
Please be aware that Metron deployment using Ambari management packs is actively being enhanced and worked upon. Some of the steps in this article and/or the behavior might be altered or become obsolete. Refer to this link for the current set of limitations with the Ambari Mpack installation and setup.
The selection of nodes, services, slaves and clients in the Ambari cluster wizard are an indicative example. Optimization for performance and scale requirements are out of scope of this article.
For Generating RPMs and Management Pack
Local system (Mac or Linux) with build tools installed viz. Maven and docker. Refer here for more details on the tool pre-requisites.
Point your browser to http://<node#1>:8080 to start the cluster installation wizard. Follow the screenshots below to complete the Metron installation.
Start the wizard by choosing the "Create a Cluster" option and specify a name for the cluster.
Select HDP 2.5 stack for Metron 0.3.x from the version selection page.
Specify the list of hosts in your cluster along with the connection information (e.g. private key)
In the "Choose Services" page, select the following services -- which are required as a minimum for a working Metron deployment. You may select other additional services as per your need.
YARN + MapReduce2
Here are sample screenshots:
In the "Assign Masters" page, ensure the following criteria is met. Observe that the Ambari wizard will display warning / error popups when the below criteria is not met.
a) Ensure that the Kibana Server, Metron Enrichment, Metron Indexing and Metron Parsers components are all assigned to the _same node_. It is important to note that these _should not_ be on the same node as Ambari server. All of these, for e.g. could reside on the node #12
b) It is preferred to run the Elasticsearch Master on the same node as Metron components (node #12, in this example).
c) Add up to 4 Kafka Broker components. Ensure that one of the Kafka Broker components is installed on the Metron node #12.
d) [Optional] You may retain the Zookeeper server on only one host and remove the rest
The other components may be left at their defaults
Note: For some of the components, there is a warning message to remind the user for client selection. Choose "Continue Anyway" if the Validation Issues warning dialog pops up.
In the "Assign Slaves and Clients" window, you need to ensure that the Metron node (#12 in this example) is selected for the following:
Optionally, you can choose to install "Client" on all the nodes.
In the "Customize Services" window, refer to the screenshots below for filling in the respective tabs
1. Change NameNode heap size from the default 1024 MB to about 4096 MB
2. Under Elasticsearch -> Advanced elastic-site -> zen_discovery_ping_unicast_hosts, specify the location where Elasticsearch master is installed (E.g. node #12 in this case.
3. Under Kibana -> Advanced kibana-env, specify the kibana_es_url to the Elasticsearch master node URL with port as 9200.
4. Under Metron, change the parameters in the respective tabs below.
Tab #1 - Default Settings for Metron services.
Provide details of the Elasticsearch hosts.
Tab #2 - Repository settings - Remote vs. Local repository
If you choose to install using a Local repository, ensure that you have copied the Metron RPMs into the /localrepo folder in the Metron node.
If you choose to install using a remote repository, specify the URL where the repo file is available.
Hit the "Next" button and then hit the "Deploy" button to proceed with Metron deployment. This will start the cluster deployment and in time, all the services should be up and running.