Community Articles

Find and share helpful community-sourced technical articles.
Labels (2)
avatar
Super Collaborator

Note:

  • This article is an extension of the instructions at this link.
  • This article uses the example of a 12-node CentOS 7 VM cluster on Openstack environment and with HDP 2.5 as base stack for Metron ver > 0.3.x (or HDP 2.4 for Metron < 0.2.x).
  • Please be aware that Metron deployment using Ambari management packs is actively being enhanced and worked upon. Some of the steps in this article and/or the behavior might be altered or become obsolete. Refer to this link for the current set of limitations with the Ambari Mpack installation and setup.
  • The selection of nodes, services, slaves and clients in the Ambari cluster wizard are an indicative example. Optimization for performance and scale requirements are out of scope of this article.

Prerequisites

For Generating RPMs and Management Pack

  • Local system (Mac or Linux) with build tools installed viz. Maven and docker. Refer here for more details on the tool pre-requisites.
  • Docker service has been installed and is running.
  • A cloned copy of the Metron git repo.

For Metron Cluster deployment

  • 12 VMs running CentOS 7.x.
  • JDK > 1.8.x installed on all nodes.
  • Ambari Server > 2.4.2.x required

Step 1 - Build Metron RPMs and Management Pack

In this step, you will create the RPMs and Ambari management pack tarball on your local system. The RPMs and Ambari mpack need to be SCP'ed into the cluster hosts

a) Build Mpack

On your local system where you have cloned the Git repo, run the following command:

cd incubator-metron
mvn clean package -DskipTests

Once the above command has run successfully, the metron management pack will be generated in the path at:

incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-xxxx.tar.gz

b) Build RPMs

Run the following commands to generate the RPMs:

Note: Docker service should be running in order for the below command to work properly.
cd incubator-metron/metron-deployment/
mvn clean package -Pbuild-rpms -DskipTests

The above command will build the RPMs and create them under incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch

c) Copy RPMs and Mpack to cluster nodes

Use either SCP or your favorite file transfer application to copy the mpack and RPMs over the to the cluster

  • Copy the metron_mpack-xxxx.tar.gz file to the node which is going to run Ambari Server (For E.g. node #1)
  • On the node where you would like to install Metron (For e.g. node 12), create a directory call /localrepo and scp all of the generated metron rpms to the /localrepo folder

Step 2 - Install Ambari Server and Metron management pack

SSH to the node where Ambari server needs to be installed (node #1) and follow these steps.

Ensure CentOS repo for Ambari server is updated:

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

Install Ambari server, install mpack and start service

yum install ambari-server -y
ambari-server setup -s
ambari-server install-mpack --mpack=/path/to/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
ambari-server start



Step 3 - Cluster Installation

Point your browser to http://<node#1>:8080 to start the cluster installation wizard. Follow the screenshots below to complete the Metron installation.

Start the wizard by choosing the "Create a Cluster" option and specify a name for the cluster.

8378-01-install-cluster.png

8379-02-name-cluster.png

Select HDP 2.5 stack for Metron 0.3.x from the version selection page.

9830-03-select-stack.png

Specify the list of hosts in your cluster along with the connection information (e.g. private key)

8381-04-specify-hosts.png

In the "Choose Services" page, select the following services -- which are required as a minimum for a working Metron deployment. You may select other additional services as per your need.

  • HDFS
  • YARN + MapReduce2
  • HBase
  • HIVE
  • Pig
  • Zookeeper
  • Storm
  • Spark
  • Kafka
  • Zeppelin
  • Elasticsearch
  • Kibana
  • Metron
  • Slider

Here are sample screenshots:

13091-05-choose-services01.png

13092-05-choose-services02.png

In the "Assign Masters" page, ensure the following criteria is met. Observe that the Ambari wizard will display warning / error popups when the below criteria is not met.

a) Ensure that the Kibana Server, Metron Enrichment, Metron Indexing and Metron Parsers components are all assigned to the _same node_. It is important to note that these _should not_ be on the same node as Ambari server. All of these, for e.g. could reside on the node #12

b) It is preferred to run the Elasticsearch Master on the same node as Metron components (node #12, in this example).

c) Add up to 4 Kafka Broker components. Ensure that one of the Kafka Broker components is installed on the Metron node #12.

d) [Optional] You may retain the Zookeeper server on only one host and remove the rest

The other components may be left at their defaults

Note: For some of the components, there is a warning message to remind the user for client selection. Choose "Continue Anyway" if the Validation Issues warning dialog pops up.

13093-06-assignmasters-warn.png

In the "Assign Slaves and Clients" window, you need to ensure that the Metron node (#12 in this example) is selected for the following:

  • DataNode
  • NodeManager
  • RegionServer
  • Supervisor
  • Flume
  • ElasticsearchDataNode
  • Client

Optionally, you can choose to install "Client" on all the nodes.

In the "Customize Services" window, refer to the screenshots below for filling in the respective tabs

1. Change NameNode heap size from the default 1024 MB to about 4096 MB8390-07-customize-namenode.png

2. Under Elasticsearch -> Advanced elastic-site -> zen_discovery_ping_unicast_hosts, specify the location where Elasticsearch master is installed (E.g. node #12 in this case.

8391-07-customize-elasticsearch-url.png

3. Under Kibana -> Advanced kibana-env, specify the kibana_es_url to the Elasticsearch master node URL with port as 9200.

8392-07-customize-kibana-es-url.png

4. Under Metron, change the parameters in the respective tabs below.

Tab #1 - Default Settings for Metron services.

Provide details of the Elasticsearch hosts.

13050-metron-screen01.png

Tab #2 - Repository settings - Remote vs. Local repository

If you choose to install using a Local repository, ensure that you have copied the Metron RPMs into the /localrepo folder in the Metron node.

9844-metron-screen2.png

If you choose to install using a remote repository, specify the URL where the repo file is available.

9845-metron-screen3.png

Hit the "Next" button and then hit the "Deploy" button to proceed with Metron deployment. This will start the cluster deployment and in time, all the services should be up and running.


03-select-stack.png982903-select-stack.png
11,730 Views
Comments

Updated to include HDP 2.5 (required for Metron > 0.3.0)

avatar
Super Collaborator

Thank you @Simon Elliston Ball, I have made some more edits as well to the screenshots.

Sir, while folloing the document i stuck over installation process of es,metron. after configuration of metron inside ambari-repository i move for metron and es installation of the nodes as recommended.But get failed with below exception:

Ambari-console error:

2017-05-2516:33:01,138-Installingpackage elasticsearch-2.3.3('/usr/bin/yum -d 0 -e 0 -y install elasticsearch-2.3.3')

2017-05-2516:33:02,132-Execution of '/usr/bin/yum -d 0 -e 0 -y install elasticsearch-2.3.3' returned 1.Error:Cannot retrieve repository metadata (repomd.xml)for repository: METRON-0.4.0.Please verify its path andtry again

2017-05-2516:33:02,132-Failed to install package elasticsearch-2.3.3.Executing'/usr/bin/yum clean metadata'

2017-05-2516:33:02,497-Retrying to install package elasticsearch-2.3.3 after 30 seconds

Command failed after 1 tries

Terminal error:

file:///localrepo/repodata/repomd.xml: [Errno 14] Could not open/read file:///localrepo/repodata/repomd.xml

Error: Cannot retrieve repository metadata (repomd.xml) for repository: METRON-0.4.0. Please verify its path and try again