Metron Indexing stopped right after started

[#1]

When i start metron indexing service via ambari, the service will start successfully but stopped after few second.

i look into /var/log/metron/ directory but only found metron-rest.log. where is the log file for other metron component?

Thanks.

[#2]

@Bramantya Anggriawan

You need to look for the indexing logs under the storm worker logs. There are two topologies that run as a part of the Metron Indexing service--random_access_indexing and batch_indexing. You can view the respective logs under /var/log/storm/worker-logs/<indexing-topo-name>/6700/worker.log.

Btw, I would also suggest that you look at the Ambari agent log for any errors. I have seen earlier that the indexing service appearing stopped could possibly be an issue with python-requests package not installed as well (see https://issues.apache.org/jira/browse/METRON-1451).

[#3]

@asubramanian

I look at ambari-agent.log and find this :

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file
    imp.load_source('__main__', script)
  File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1.1.4.1.0/package/scripts/indexing_master.py", line 18, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 8, in <module>
    from .connectionpool import (
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 11, in <module>
    from .exceptions import (
  File "/usr/lib/python2.7/site-packages/urllib3/exceptions.py", line 2, in <module>
    from .packages.six.moves.http_client import (
  File "/usr/lib/python2.7/site-packages/urllib3/packages/six.py", line 198, in load_module
    return sys.modules[fullname]
AttributeError: 'NoneType' object has no attribute 'modules'

I already update the python-request but it looks like the problem is python-urllib3 now.

[#4]

Can you post the version of python-requests you have installed?

pip list | grep requests

[#5]

Also, can you describe your test environment more...

* full dev or bare metal

* version of Metron

[#6]

@asubramanian

here is the result :

[root@workernode-01 ~]# pip list | grep requests
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
requests (2.18.4)
[root@workernode-01 ~]# 

my cluster build on vm with RHIF bare metal, with metron version 0.4.1.1.4.1.0

[#7]

ok.. can you install the following version of requests and check if things work?

pip install requests==2.6.1

[#8]

Such a great post to know different useful information.

Logomines

[#9]

@asubramanian

after i install request 2.6.1 like your advice, this error still persist :

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file
    imp.load_source('__main__', script)
  File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1.1.4.1.0/package/scripts/indexing_master.py", line 18, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 3, in <module>
    from . import urllib3
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 61, in load_module
AttributeError: 'NoneType' object has no attribute 'modules'

[#10]

Hm.. this error seems different than the earlier one. Can you list the version of urllib3 ? (pip list | grep url)

[#11]

@asubramanian

below are urllib3 :

[root@workernode-01 ~]# pip list | grep url
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pycurl (7.19.0)
urlgrabber (3.10)
urllib3 (1.22)

[#12]

can you find solution for this problem ?

[#13]

Have you checked this issue with python-requests package not installed: https://issues.apache.org/jira/browse/METRON-1451 ?

[#14]

Hi @asubramanian ,

Yes , I installed python-requests but nothing changed.

Python module version that installed on server :

urllib3==1.24.1
pyOpenSSL==18.0.0
requests==2.6.1

I have this problem with HCP 1.7.1 On CentOS 7.

[#15 · Accepted solution]

It actually running after i restart the server