Community Articles

Find and share helpful community-sourced technical articles.
Celebrating as our community reaches 100,000 members! Thank you!
Labels (3)
Super Guru

In Hortworks HCP they use an ElasticSearch Mpack for version 5.x of Elasticsearch and Kibana:

I have taken this and expanded it to include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat.


My cluster is 6 nodes.

ElasticSearch is installed on Nodes 4,5,6(4 Master & 5,6 Data Nodes).
Logstash is Installed on Node 3.
FileBeat & MetricBeat are installed on all 6 nodes.
Kibana is installed on Node 4.
The rest of the cluster is configured normally for a minimal Install.


Orginal HCP Mpack


My MPack 6.3.0 - Version Change + Logstash + Beats


My Mpack 6.3.2 - Version Change + Logstash (multi-node & jvm settings config) + sudo non root capabilities


Installation Steps

1. Deliver Management Pack tar.gz to local filesystem on Ambari-Server

upload to hdfs view Files View

download to ambari-server node (/home/root)

2. Install Management Pack

sudo ambari-server install-mpack --mpack=/home/root/elasticsearch_mpack- --verbose

Uninstall is: sudo ambari-server uninstall-mpack --mpack-name=elasticsearch-ambari.mpack

3. Restart Ambari Server

sudo ambari-server restart

4. Use Ambari Add Service to Install ELK Stack Components

The following settings will be required during the Install Wizard




ElasticSearch Zen Discovery Hosts:
example: [,, ]


Post installation Ambari handles the configuration of all components including logstash (input, output, and filters) and FileBeat and MetricBeat configuration files.

  • ElasticSearch configuration should work out of the box without any changes other than Zen Discovery Hosts.
  • Logstash filters are setup for beats input, file FileBeat filter, and elasticsearch output.
  • FileBeat is setup to use Logstash.
  • MetricBeat is setup to send metrics directly to Elasticsearch.


Learning this Elasticsearch MPack is a good example of how to create your own custom stack using a Management Pack to define services not normally found in an Ambari Cluster. Ambari Administrators looking to understand how to create their own Management Pack should take some time to diff the Mpacks attached below. Creating custom services controlled via Ambari is pretty easy if you mimic the folder structure, make the necessary xml file changes, and adjust the python package scripts accordingly.

0 Kudos
Rising Star

@ Steve Matison

Thank you for posting the ES MPACK. Would you be able to share how to build our own custom MPACK. Maybe a follow on blog.



Super Guru

@Amit Nandi sure I will work on an article for how to make an mpack.

Super Guru

Today I am working on an ELK Mpack for HDP 3.0+ and HDF 3.0+. To pick this back up I have started on another documentation journey to record my progress. Starting with this post and I was able to get a base cluster up with the ELK above. I will be creating another article for getting this to work on the new HDP stacks as well as changes for most recent version of ELK stack.


When i try to install ELK to my cluster (HDP  3.0, HDF 3.2 and Ambari 2.7.1) according to these steps, i get this error:


Traceback (most recent call last):
File "/usr/sbin/", line 1060, in <module>
File "/usr/sbin/", line 1030, in mainBody
main(options, args, parser)
File "/usr/sbin/", line 980, in main
File "/usr/sbin/", line 79, in execute
self.fn(*self.args, **self.kwargs)
File "/usr/lib/ambari-server/lib/ambari_server/", line 900, in install_mpack
(mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode)
File "/usr/lib/ambari-server/lib/ambari_server/", line 700, in _install_mpack
tmp_root_dir = expand_mpack(tmp_archive_path)
File "/usr/lib/ambari-server/lib/ambari_server/", line 151, in expand_mpack
archive_root_dir = get_archive_root_dir(archive_path)
File "/usr/lib/ambari-server/lib/resource_management/libraries/functions/", line 82, in get_archive_root_dir
with closing(, mode(archive))) as tar:
File "/usr/lib64/python2.7/", line 1678, in open
return func(name, filemode, fileobj, **kwargs)
File "/usr/lib64/python2.7/", line 1729, in gzopen
raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file


Does anyone know the cause of this error?

Super Guru

@mervezeybel  this error is caused by sending the mpack command to ambari but without a valid mpack file.   Not sure what full command you gave but I have seen this happen before when the url I use is wrong.   Couple of things you can do:


Try to wget the url to a local file, then adjust command to use the local file.


If the mpack url you are using is still not working,  try to get it again from my GitHub directly:


There is a newer version there as well.      Make sure you get the correct /raw/ link if using the github links (see sample below).  If you just take the /blob/ link straight from the page, it will result in same error you have (not a gzip file).


ambari-server install-mpack --mpack= --verbose



Thank you @stevenmatison. I can install properly from

New Contributor

I try to install elasticsearch-6.4.2 to my cluster(HDP 3.1 Ambari 2.7.3) , intallation was completed successfully but it could not start, and the error encounterd:


Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/", line 168, in <module>
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/", line 352, in execute
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/", line 1011, in restart
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/", line 153, in start
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/", line 86, in configure
  File "/usr/lib/ambari-agent/lib/resource_management/core/", line 166, in __init__
  File "/usr/lib/ambari-agent/lib/resource_management/core/", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/", line 124, in run_action
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/", line 123, in action_create
    content = self._get_content()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/", line 160, in _get_content
    return content()
  File "/usr/lib/ambari-agent/lib/resource_management/core/", line 52, in __call__
    return self.get_content()
  File "/usr/lib/ambari-agent/lib/resource_management/core/", line 144, in get_content
    rendered = self.template.render(self.context)
  File "/usr/lib/ambari-agent/lib/ambari_jinja2/", line 891, in render
    return self.environment.handle_exception(exc_info, True)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/templates/elasticsearch.master.yml.j2", line 93, in top-level template code
    action.destructive_requires_name: {{action_destructive_requires_name}}
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/", line 73, in __getattr__
    raise Fail("Configuration parameter '" + + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hostname' was not found in configurations dictionary!


I modified the property of from elastic-config.xml and hostname from elasticsearch-env.xml, However, it still could not start  and the same error encountered, do you have any idea?