question Re: Beeline connection hangs without any error in Archives of Support Questions (Read Only)

Beeline connection hangs without any error

khanarshadazmat — Mon, 14 May 2018 14:02:09 GMT

My HDP version : HDP-2.3.4.0 and Ambari : Version2.5.2.0

I'm trying to connect beeline through my edge node using string :

beeline -n abcd -u jdbc:hive2://xyz.abc.org:10001/default;ssl=true;transportMode=http;httpPath=cliservice

but its not proceeding getting hung without any error :

root@had02:~# beeline -n abcd -u jdbc:hive2://xyz.abc.org:10001/default;ssl=true;transportMode=http;httpPath=cliservice WARNING: Use "yarn jar" to launch YARN applications. Connecting to jdbc:hive2://xyz.abc.org:10001/default

whereas I check service is running from webui and 10001 port is listing and checked hiveserver2 logs as well but no issue reported please suggest me any solution or where can I identify this issue.

Re: Beeline connection hangs without any error

umair_khan — Mon, 14 May 2018 21:05:20 GMT

Check firewall on all servers.

Re: Beeline connection hangs without any error

Shelton — Tue, 15 May 2018 01:04:52 GMT

@Arshadullah Khan

To use SSL for Hiveserver2, you will need to first enable SSL for Hiveserver2. If you have already done that then the connect string should look like below

beeline -n abcd -u jdbc:hive2://xyz.abc.org:10001/default;ssl=true;sslTrustStore=/path/to/truststore.jks;trustStorePassword=Password

Hope that helps

Re: Beeline connection hangs without any error

khanarshadazmat — Tue, 29 May 2018 20:26:59 GMT

Hi Guys,

Thanks for the reply.

After analyzing Hiveserver logs we found that Hiveserver was unable to create new threads and hence Hive queries are not working.

This is an intermittent issue. Whenever thread limit reached, Hiveserver is not able to create a new connection and stops responding. After restarting Hiveserver, it releases all threads and hive starts responding again.

Here is the stack trace:

     
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:791)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
        at org.apache.hadoop.ipc.Client.call(Client.java:1397)
        ... 73 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:784)
        ... 76 more

We found the similar issue at Hortonworks community:
https://community.hortonworks.com/content/supportkb/151804/error-javalangoutofmemoryerror-unable-to-create-ne.html

According to Hortonworks, its a bug in HDP-2.3.4 version and it got resolved in a newer version of HDP.