Description Use this Mpack to install ELK 6.3.2         Repo Info: DFHeinz ELK Stack Mpack for Ambari. Install Elasticsearch, Logstash, Kibana, FileBeat, & MetricBeat with Ambari using this management pack created by DFHeinz.  Repo URL:        https://github.com/steven-dfheinz/dfhz_elk_mpack   Account Name:  steven-dfheinz   Repo Name: dfhz_elk_mpack   ... View more

This is a Work In Progress Article. Downloads: HDP/HDF 3 Ambari 2.7 Mpack ELK 6.3.2 with sudo (for non root user install) elasticsearch_mpack-3.0.0.0-1.tar.gz HDP/HDF 3 Ambari 2.7 Mpack ELK 6.3.2 without sudo elasticsearch_mpack-3.0.0.0-0.tar.gz In the Parent Article I introduce the process to take an existing HortonWorks ELK Mpack and take it through a series of versions (up to 2.6) which would allow me to install the ELK Version and Components I want (ElasticSearch, Logstash, Kibana, FileBeats, MetricBeats) in Ambari 2.6. In this article I am going to install the last articles ELK 2.6 version into my local machine using Vagrant. With this working Test Base I will version the Mpack up to 3.0 as I change the files to allow install into HDP & HDF 3.0. I am also going to change to the current version of ELK 6.6.1. Starting with the 2.6 Mpack and the Ambari Quick Start Guide I am able to get a cluster installed very easily in my local machine. In my Test Base install I chose a single node c7401.ambari.apache.org. For the purpose of this Test Base I only want to complete the most minimal install to support the Mpack Services without any issues during the Install Wizard. On this single node I install Ambari Server and Agent and the following components: Zookeeper Ambari Metrics ElasticSearch LogStash Kibana FileBeats MetricBeats Terminal Commands Required In Local Machine git clone https://github.com/u39kun/ambari-vagrant.git sudo -s 'cat ambari-vagrant/append-to-etc-hosts.txt >> /etc/hosts' cd ambari-vagrant/centos7.4 cp ../insecure_private_key . cp ~/Downloads/elasticsearch_mpack-2.6.0.0-9.tar.gz . vagrant up c7401 vagrant ssh c7401 Terminal Commands Required in Vagrant Node wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.1.0/ambari.repo yum --enablerepo=extras install epel-release -y yum install java java-devel ambari-server ambari-agent -y ambari-server setup -s ambari-server install-mpack --mpack=/vagrant/elasticsearch_mpack-2.6.0.0-9.tar.gz --verbose ambari-server start ambari-agent start After install there were some issues with starting the services. This is okay for now, the ELK Mpack really needs 4 nodes. Most of the work for versioning will be making sure the new Stack Versions are included. I can test all of these changes without expecting any services to work. With the Test Base complete I can now quickly spin up fresh clusters working my way from HDP 2.6 to HDP 3.1.0 and ELK 6.3.2 to 6.6.1. The ELK versions will likely require some additional configuration changes, so this version will be completed in a final test base that will include HDP 3.1.0, 4 nodes, and expect the services to start. Results From Using This Test Base HDP It took me a few sessions working with this base to figure out that the out of box install issues for HDP 3 were related to just a few conflicts in the original Mpack parameters. Conflict with config settings for user management. The following python command was necessary: python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n DFHZ_ELK -l c7401.ambari.apache.org -t 8080 -a set -c cluster-env -k  ignore_groupsusers_create -v true Adjustments to Mpack services params.py to get hostname and java_home from slightly different paths in the config object. hostname = config['agentLevelParams']['hostname'] java64_home = config['ambariLevelParams']['java_home'] With a working install for this new Mpack in HDP 3 I can now start working on a 2 node cluster to make sure nothing else is required to get the original 4 node ELK stack installed on HDP 3. After the 2 node install cluster wizard is complete I did have to manually start some ELK services. HDF 2 Node Test Base Next I need to create an HDF cluster and make sure this Mpack works there. This requires a fully new test base and some changes to the mpack.json file to include HDF stack_version. Terminal Command Required in Vagrant Node for Ambari Master wget http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.3.1.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.3.1.0-10.tar.gz && wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel ambari-server ambari-agent -y && ambari-server setup -s && ambari-server install-mpack --mpack=/root/hdf-ambari-mpack-3.3.1.0-10.tar.gz --verbose && ambari-server install-mpack --mpack=/vagrant/elasticsearch_mpack-3.0.0.0-0.tar.gz --verbose && ambari-server start && ambari-agent start Terminal Command Required in Vagrant Node Ambari Agent wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel ambari-agent -y && ambari-agent start Results From Using Test Base HDF It took me quite a few attempts to identify a solution to very many symptoms. Somewhere in the config object for HDF are some non UTF-8 characters. In some places this threw and error, in other places it ended up silently creating empty files across the elk stack during install. I added these lines to the python scripts: # encoding=utf8 import sys reload(sys) sys.setdefaultencoding('utf8') Once I identified the solution for the component python scripts I was able to get the stack installed without errors and running in HDF with only some starting issues. After starting components manually, Logstash & Kibana reporting as stopped but they are actually running. During my next sitting I focused on the stopped services. I set the Ambari-agent log level to DEBUG and noticed some additional terminal output in the command status output from "sudo service start". After changing Kibana and Logstash to just service start, I was able to manually stop the services in the node, then start from Ambari. I am not 100% sure this sudo was related. At any rate the ELK Mpack is now installed and all services running in HDF: I am going to go through a few more complete tests to make sure I can get the cluster stable right after install without any additional work. I completed my final test with "sudo service" replaced with "service". During Cluster Install Wizard everything installed without errors. The only issue was a warning for Check ElasticSearch which happens faster than ElasticSearch is started up. I came back to Ambari w/ Elasticsearch, Kibana, and Logstash running. I just had to manually start FileBeats and Metricbeats. Now I will be able to focus on last part of this article: upgrade ELK 6.3.1 to 6.6.1. ... View more

In Hortworks HCP they use an ElasticSearch Mpack for version 5.x of Elasticsearch and Kibana: http://public-repo-1.hortonworks.com/HCP/centos7/1.x/updates/1.6.0.0/tars/metron/elasticsearch_mpack-1.6.0.0-7.tar.gz I have taken this and expanded it to include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat. My cluster is 6 nodes. ElasticSearch is installed on Nodes 4,5,6(4 Master & 5,6 Data Nodes). Logstash is Installed on Node 3. FileBeat & MetricBeat are installed on all 6 nodes. Kibana is installed on Node 4. The rest of the cluster is configured normally for a minimal Install. Downloads Orginal HCP Mpack elasticsearch-mpack-1600-7.tar.gz My MPack 6.3.0 - Version Change + Logstash + Beats elasticsearch-mpack-2500-9.tar.gz My Mpack 6.3.2 - Version Change + Logstash (multi-node & jvm settings config) + sudo non root capabilities elasticsearch-mpack-2600-9.tar.gz Installation Steps 1. Deliver Management Pack tar.gz to local filesystem on Ambari-Server upload to hdfs view Files View download to ambari-server node (/home/root) 2. Install Management Pack sudo ambari-server install-mpack --mpack=/home/root/elasticsearch_mpack-2.6.0.0-9.tar.gz --verbose Uninstall is: sudo ambari-server uninstall-mpack --mpack-name=elasticsearch-ambari.mpack 3. Restart Ambari Server sudo ambari-server restart 4. Use Ambari Add Service to Install ELK Stack Components The following settings will be required during the Install Wizard ES_URL: example: http://node4.hostname.com:9200 KIBANA_URL: example: http://node4.hostname.com:5000 LOGSTASH_URL: example: http://node3.hostname.com:5044 ElasticSearch Zen Discovery Hosts: example: [ node4.hostname.com, node5.hostname.com, node6.hostname.com ] Configuration Post installation Ambari handles the configuration of all components including logstash (input, output, and filters) and FileBeat and MetricBeat configuration files. ElasticSearch configuration should work out of the box without any changes other than Zen Discovery Hosts. Logstash filters are setup for beats input, file FileBeat filter, and elasticsearch output. FileBeat is setup to use Logstash. MetricBeat is setup to send metrics directly to Elasticsearch. Summary Learning this Elasticsearch MPack is a good example of how to create your own custom stack using a Management Pack to define services not normally found in an Ambari Cluster. Ambari Administrators looking to understand how to create their own Management Pack should take some time to diff the Mpacks attached below. Creating custom services controlled via Ambari is pretty easy if you mimic the folder structure, make the necessary xml file changes, and adjust the python package scripts accordingly. ... View more