Metron Indexing topology appears stopped in Ambari UI

[#1]

Problem

On some of the HCP deployments, the Indexing topology might show up as "stopped, while the actual topology might be running (when you check in Storm UI).

Additionally, one might also see the following kind of error messages in ambari-agent,log.

INFO 2018-02-05 22:21:39,990 PythonReflectiveExecutor.py:67 - Reflective command failed with exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file
    imp.load_source('__main__', script)
  File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/package/scripts/indexing_master.py", line 18, in <module>
    import requests
  File "/usr/lib/python2.6/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/usr/lib/python2.6/site-packages/requests/packages/__init__.py", line 61, in load_module
    if name in sys.modules:
AttributeError: 'NoneType' object has no attribute 'modules'

Reasoning

• If the above symptoms are true, you are most likely seeing METRON-1451

Solution

1. Install version 2.61. of python-requests on the server pip install requests==2.6.1
2. Restart the Indexing topology to resolve the issue.

[#2]

Hello,

Still getting below error. Still not able to start Metron Indexing.

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures}

Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file

imp.load_source('__main__', script)

File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1.1.4.2.0/package/scripts/indexing_master.py", line 18, in <module>

import requests

File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>

from .packages.urllib3.contrib import pyopenssl

File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 3, in <module>

from . import urllib3

File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 61, in load_module

AttributeError: 'NoneType' object has no attribute 'modules'

[#3]

Did you restart the Indexing topology after the change? Note that you might have to restart from command line, since Ambari will not allow stopping of an (already) stopped service.

Can you post the version of python-requests you are running?

[#4]

I encountered similar problem with EL7 and python requests=2.6.0, fixed as below:

[root@XXXX ~]# yum remove python-requests-2.6.0-1.el7_1.noarch
Loaded plugins: langpacks, product-id, search-disabled-repos
Repository HDP-UTILS-1.1.0.21 is listed more than once in the configuration
Resolving Dependencies
Installed size: 774 k
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : insights-client-3.0.3-9.el7_5.noarch                                                                                                                                                        1/2
warning: /etc/insights-client/insights-client.conf saved as /etc/insights-client/insights-client.conf.rpmsave
  Erasing    : python-requests-2.6.0-1.el7_1.noarch                                                                                                                                                        2/2
  Verifying  : python-requests-2.6.0-1.el7_1.noarch                                                                                                                                                        1/2
  Verifying  : insights-client-3.0.3-9.el7_5.noarch                                                                                                                                                        2/2
Removed:
  python-requests.noarch 0:2.6.0-1.el7_1
Dependency Removed:
  insights-client.noarch 0:3.0.3-9.el7_5
Complete!
[root@XXXX ~]# pip install requests
Collecting requests
  Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925df... (91kB)
    100% |████████████████████████████████| 92kB 5.2MB/s
Requirement already satisfied: idna<2.8,>=2.5 in /usr/lib/python2.7/site-packages (from requests) (2.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python2.7/site-packages (from requests) (3.0.4)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in /usr/lib/python2.7/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python2.7/site-packages (from requests) (2018.4.16)
Installing collected packages: requests
Successfully installed requests-2.19.1