Hive HiveServer2 Interactive LLAP doesn't start

[#1]

I use HDP 2.6.3, no kerberos, no ssl. Hive LLAP doesn't start.

I tried many suggestions including llap_heap_size and hive.llap.daemon.yarn.container.mb change, JAVA_HOME change etc. but nothing worked.

ERRORs from YARN LLAP appllication logs:

ERROR 2018-06-04 09:32:30,173 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:32:30,173 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:32:30,175 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:32:40,185 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:32:40,290 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:32:40,290 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:32:40,292 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:32:50,298 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:32:50,408 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:32:50,408 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:32:50,410 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:00,422 connection.py:573 - Connecting to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:00,424 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:33:00,428 connection.py:540 - Closing connection to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:00,428 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:33:00,428 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:33:00,429 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:00,429 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:00,429 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:00,531 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:00,532 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:00,534 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:10,537 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:10,633 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:10,634 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:10,636 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:20,636 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:20,735 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:20,736 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:20,738 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:30,743 connection.py:573 - Connecting to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:30,745 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:33:30,748 connection.py:540 - Closing connection to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:30,748 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:33:30,749 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:33:30,749 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:30,749 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:30,749 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:30,848 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:30,849 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:30,851 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:40,861 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:40,964 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:40,964 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:40,966 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:50,976 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:51,076 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:51,076 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:51,078 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:01,081 connection.py:573 - Connecting to analitik03.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:01,084 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:34:01,087 connection.py:540 - Closing connection to analitik03.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:01,087 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:34:01,088 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:34:01,088 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:01,088 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:01,089 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:01,186 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:01,186 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:01,188 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:11,197 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:11,286 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:11,286 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:11,288 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:21,298 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:21,400 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:21,400 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:21,402 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:31,414 connection.py:573 - Connecting to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:31,417 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:34:31,420 connection.py:540 - Closing connection to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:31,420 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:34:31,420 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:34:31,421 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:31,421 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:31,421 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:31,520 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:31,521 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:31,523 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:41,533 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:41,634 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:41,634 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:41,636 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:51,647 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:51,745 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:51,745 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:51,747 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:35:01,759 connection.py:573 - Connecting to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:35:01,761 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:35:01,765 connection.py:540 - Closing connection to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:35:01,765 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:35:01,765 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:35:01,766 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:35:01,766 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:35:01,766 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:35:01,865 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:35:01,865 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:35:01,867 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:35:11,877 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:35:11,979 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:35:11,979 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:35:11,981 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:35:21,991 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:35:22,093 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:35:22,093 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions. 
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:35:22,094 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...

On Ambari interface:

2018-06-04 09:56:26,643 - LLAP app 'llap0' deployment unsuccessful.
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 626, in <module>
    HiveServerInteractive().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 978, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 119, in start
    raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.")
resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.

SupportKB handles the same issue and asserts that he has solved the problem which is not.

[#2]

@Erkan ŞİRİN

Your real problem is "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed]" On an unsecure hadoop cluster with python 2.7.9 installed, slider agent fails with SSL validation errors

Check if your python version is python 2.7.9 and Slider version is less than 0.92

https://issues.apache.org/jira/browse/SLIDER-942

If the above is correct then download the patch and try

[#3]

Thank you @Geoffrey Shelton Okot. slider 0.92 and python 2.7.5.

[#4 · Accepted solution]

Initially SupportKB recomended solution doesn't work. But then I have realized my amazing mistake:

echo -e "[https]\nverify=disable" > /etc/python/cert-verification.cfg 

command somehow generated:

[https]nverify=disable 

which should be:

[https]
verify=disable

After fixing this wierd mistake LLAP successfully started. Thanks @Geoffrey Shelton Okot for your amazing response.

[#5]

Great to know your LLAP started !!!

[#6]

@Erkan ŞİRİN

Some deep dive in the setup could be worthy,please have a look at these 2 resources, they could be of help.

https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html

https://community.hortonworks.com/articles/149899/investigating-when-llap-doesnt-start.html