Metron HCP Cybersecurity installation fails using Ambari due to Kibana failure

[#1]

Hi, please help with HCP Metron installation failure using Ambari. I tried both CentOS 6 and CentOS 7, tried HDP 2.5 and HDP 2.6 with the latest version of HCP 1.4.0.

Every time Ambari fails to install it because of Kibana problem, which fails on this step: Failed to install package python-elasticsearch.

Please suggest anything. I spent several days on this trying different stack and OS versions. Here is the full log of Kibana install:

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py", line 120, in <module> Kibana().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env)
  File "/var/lib/ambari-agent/cache/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py", line 42, in install self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 821, in install_packages retry_count=agent_stack_retry_count)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 53, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 264, in install_package self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries return self._call_with_retries(cmd, is_checked=True, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries code, out = func(cmd, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install python-elasticsearch' returned 1. Error: Nothing to do stdout:
2018-01-31 02:34:43,033 - Stack Feature Version Info: Cluster Stack=2.5, Command Stack=None, Command Version=None -> 2.5
2018-01-31 02:34:43,038 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-01-31 02:34:43,039 - Group['metron'] {}
2018-01-31 02:34:43,040 - Group['livy'] {}
2018-01-31 02:34:43,040 - Group['elasticsearch'] {}
2018-01-31 02:34:43,040 - Group['spark'] {}
2018-01-31 02:34:43,041 - Group['hdfs'] {}
2018-01-31 02:34:43,041 - Group['zeppelin'] {}
2018-01-31 02:34:43,041 - Group['hadoop'] {}
2018-01-31 02:34:43,041 - Group['kibana'] {}
2018-01-31 02:34:43,041 - Group['users'] {}
2018-01-31 02:34:43,041 - Group['knox'] {}
2018-01-31 02:34:43,042 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,043 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,044 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,045 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,046 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-01-31 02:34:43,047 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,048 - User['elasticsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,049 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,050 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,051 - User['kibana'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,052 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,052 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,053 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,054 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,055 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-01-31 02:34:43,056 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-01-31 02:34:43,057 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'zeppelin', u'hadoop'], 'uid': None}
2018-01-31 02:34:43,058 - User['metron'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,059 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,060 - User['mahout'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,061 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-01-31 02:34:43,062 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,063 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs'], 'uid': None}
2018-01-31 02:34:43,064 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,065 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,066 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,067 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-01-31 02:34:43,067 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-01-31 02:34:43,069 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-01-31 02:34:43,073 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-01-31 02:34:43,074 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-01-31 02:34:43,074 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-01-31 02:34:43,076 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-01-31 02:34:43,076 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2018-01-31 02:34:43,083 - call returned (0, '1011')
2018-01-31 02:34:43,084 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1011'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-01-31 02:34:43,089 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1011'] due to not_if
2018-01-31 02:34:43,089 - Group['hdfs'] {}
2018-01-31 02:34:43,089 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', u'hdfs']}
2018-01-31 02:34:43,090 - FS Type: 
2018-01-31 02:34:43,090 - Directory['/etc/hadoop'] {'mode': 0755}
2018-01-31 02:34:43,103 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-01-31 02:34:43,104 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-01-31 02:34:43,117 - Repository['ES-Curator-5.x-repo-1'] {'append_to_file': False, 'base_url': 'http://packages.elastic.co/curator/5/centos/7', 'action': ['create'], 'components': [u'CURATOR', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-01-31 02:34:43,123 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[ES-Curator-5.x-repo-1]\nname=ES-Curator-5.x-repo-1\nbaseurl=http://packages.elastic.co/curator/5/centos/7\n\npath=/\nenabled=1\ngpgcheck=0'}
2018-01-31 02:34:43,124 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-01-31 02:34:43,124 - Repository['HCP-1.4.0.0-38-repo-1'] {'append_to_file': True, 'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HCP/centos6/1.x/BUILDS/1.4.0.0-38', 'action': ['create'], 'components': [u'METRON', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-01-31 02:34:43,127 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[ES-Curator-5.x-repo-1]\nname=ES-Curator-5.x-repo-1\nbaseurl=http://packages.elastic.co/curator/5/centos/7\n\npath=/\nenabled=1\ngpgcheck=0\n[HCP-1.4.0.0-38-repo-1]\nname=HCP-1.4.0.0-38-repo-1\nbaseurl=http://s3.amazonaws.com/dev.hortonworks.com/HCP/centos6/1.x/BUILDS/1.4.0.0-38\n\npath=/\nenabled=1\ngpgcheck=0'}
2018-01-31 02:34:43,127 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-01-31 02:34:43,128 - Repository['HDP-2.5-repo-1'] {'append_to_file': True, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-01-31 02:34:43,130 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[ES-Curator-5.x-repo-1]\nname=ES-Curator-5.x-repo-1\nbaseurl=http://packages.elastic.co/curator/5/centos/7\n\npath=/\nenabled=1\ngpgcheck=0\n[HCP-1.4.0.0-38-repo-1]\nname=HCP-1.4.0.0-38-repo-1\nbaseurl=http://s3.amazonaws.com/dev.hortonworks.com/HCP/centos6/1.x/BUILDS/1.4.0.0-38\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-2.5-repo-1]\nname=HDP-2.5-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0\n\npath=/\nenabled=1\ngpgcheck=0'}
2018-01-31 02:34:43,131 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-01-31 02:34:43,131 - Repository['HDP-UTILS-1.1.0.21-repo-1'] {'append_to_file': True, 'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-01-31 02:34:43,134 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': ...}
2018-01-31 02:34:43,134 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-01-31 02:34:43,134 - Repository['elasticsearch-5.x-repo-1'] {'append_to_file': True, 'base_url': 'https://artifacts.elastic.co/packages/5.x/yum', 'action': ['create'], 'components': [u'ELASTICSEARCH', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-01-31 02:34:43,137 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': ...}
2018-01-31 02:34:43,137 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-01-31 02:34:43,138 - Repository['kibana-5.x-repo-1'] {'append_to_file': True, 'base_url': 'https://artifacts.elastic.co/packages/5.x/yum', 'action': ['create'], 'components': [u'KIBANA', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-01-31 02:34:43,140 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': ...}
2018-01-31 02:34:43,140 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-01-31 02:34:43,141 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-01-31 02:34:43,212 - Skipping installation of existing package unzip
2018-01-31 02:34:43,212 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-01-31 02:34:43,220 - Skipping installation of existing package curl
2018-01-31 02:34:43,220 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-01-31 02:34:43,227 - Skipping installation of existing package hdp-select
2018-01-31 02:34:43,231 - The repository with version 2.5.3.0-37 for this command has been marked as resolved. It will be used to report the version of the component which was installed
2018-01-31 02:34:43,236 - Skipping stack-select on KIBANA because it does not exist in the stack-select package structure.
2018-01-31 02:34:43,443 - Installing Kibana
2018-01-31 02:34:43,443 - Package['python-elasticsearch'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-01-31 02:34:43,515 - Installing package python-elasticsearch ('/usr/bin/yum -d 0 -e 0 -y install python-elasticsearch')
2018-01-31 02:34:45,804 - Execution of '/usr/bin/yum -d 0 -e 0 -y install python-elasticsearch' returned 1. Error: Nothing to do
2018-01-31 02:34:45,804 - Failed to install package python-elasticsearch. Executing '/usr/bin/yum clean metadata'
2018-01-31 02:34:45,987 - Retrying to install package python-elasticsearch after 30 seconds
2018-01-31 02:35:24,067 - The repository with version 2.5.3.0-37 for this command has been marked as resolved. It will be used to report the version of the component which was installed
2018-01-31 02:35:24,073 - Skipping stack-select on KIBANA because it does not exist in the stack-select package structure.


Command failed after 1 tries

[#2 · Accepted solution]

Did you install epel-relase repo ? ( yum install -y epel-release)

[root@ip-172-31-21-188 ~]# yum --disablerepo=* --enablerepo=epel search python-elasticsearch
Loaded plugins: fastestmirror
epel/x86_64/metalink                                                                                                                                                                                                  |  15 kB  00:00:00
epel                                                                                                                                                                                                                  | 4.7 kB  00:00:00
(1/3): epel/x86_64/group_gz                                                                                                                                                                                           | 266 kB  00:00:00
(2/3): epel/x86_64/updateinfo                                                                                                                                                                                         | 879 kB  00:00:00
(3/3): epel/x86_64/primary_db                                                                                                                                                                                         | 6.2 MB  00:00:00
Determining fastest mirrors
 * epel: s3-mirror-eu-west-1.fedoraproject.org
===================================================================================================== N/S matched: python-elasticsearch =====================================================================================================
python-elasticsearch.noarch : Client for Elasticsearch


  Name and summary matches only, use "search all" for everything.

[#3]

It works!

Thanks for your info.

yum install -y epel-release