Support Questions

Find answers, ask questions, and share your expertise

Beeline connection hangs without any error

avatar
Contributor

My HDP version : HDP-2.3.4.0 and Ambari : Version2.5.2.0

I'm trying to connect beeline through my edge node using string :

beeline -n abcd -u jdbc:hive2://xyz.abc.org:10001/default;ssl=true;transportMode=http;httpPath=cliservice

but its not proceeding getting hung without any error :

root@had02:~# beeline -n abcd -u jdbc:hive2://xyz.abc.org:10001/default;ssl=true;transportMode=http;httpPath=cliservice WARNING: Use "yarn jar" to launch YARN applications. Connecting to jdbc:hive2://xyz.abc.org:10001/default

whereas I check service is running from webui and 10001 port is listing and checked hiveserver2 logs as well but no issue reported please suggest me any solution or where can I identify this issue.

1 ACCEPTED SOLUTION

avatar
Contributor

Hi Guys,

Thanks for the reply.

After analyzing Hiveserver logs we found that Hiveserver was unable to create new threads and hence Hive queries are not working.

This is an intermittent issue. Whenever thread limit reached, Hiveserver is not able to create a new connection and stops responding. After restarting Hiveserver, it releases all threads and hive starts responding again.

Here is the stack trace:

     
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:791)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
at org.apache.hadoop.ipc.Client.call(Client.java:1397)
... 73 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:784)
... 76 more

We found the similar issue at Hortonworks community:
https://community.hortonworks.com/content/supportkb/151804/error-javalangoutofmemoryerror-unable-to-...

According to Hortonworks, its a bug in HDP-2.3.4 version and it got resolved in a newer version of HDP.

View solution in original post

3 REPLIES 3

avatar
Expert Contributor

Check firewall on all servers.

avatar
Master Mentor

@Arshadullah Khan

To use SSL for Hiveserver2, you will need to first enable SSL for Hiveserver2. If you have already done that then the connect string should look like below

beeline -n abcd -u jdbc:hive2://xyz.abc.org:10001/default;ssl=true;sslTrustStore=/path/to/truststore.jks;trustStorePassword=Password

Hope that helps

avatar
Contributor

Hi Guys,

Thanks for the reply.

After analyzing Hiveserver logs we found that Hiveserver was unable to create new threads and hence Hive queries are not working.

This is an intermittent issue. Whenever thread limit reached, Hiveserver is not able to create a new connection and stops responding. After restarting Hiveserver, it releases all threads and hive starts responding again.

Here is the stack trace:

     
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:791)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
at org.apache.hadoop.ipc.Client.call(Client.java:1397)
... 73 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:784)
... 76 more

We found the similar issue at Hortonworks community:
https://community.hortonworks.com/content/supportkb/151804/error-javalangoutofmemoryerror-unable-to-...

According to Hortonworks, its a bug in HDP-2.3.4 version and it got resolved in a newer version of HDP.