[Cloudera][HiveJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: Socket is closed by peer.

[#1]

hi community cloudera, 

i'm experiencing strange hive drive behaviour:

my app consequently performing `select` queries on multiple tables successfully. But at some point it gets an error:

java.sql.SQLException: [Cloudera][HiveJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: Socket is closed by peer. FetchResults for query "SELECT `iq***".

and when the app retries to reconnect (because java.sql.Connection.isValid() == false)

it constantly gets 

[Cloudera][JDBC](10060) Connection has been closed.

java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10060) Connection has been closed.

at com.cloudera.hiveserver2.exceptions.ExceptionConverter.toSQLException(Unknown Source)

at com.cloudera.hiveserver2.jdbc.common.SConnection.closeConnection(Unknown Source)

at com.cloudera.hiveserver2.jdbc.common.SConnection.abortInternal(Unknown Source)

at com.cloudera.hiveserver2.jdbc.common.SConnection.close(Unknown Source)

Since connection string is 

jdbc:hive2://hive2host.loc:10000;AuthMech=3

I don't have access to Hive logs, so I can guess only.

Could it be relates to LDAP server issues? Or it is by some other reason? 

[#2]

@denysobukhov, Welcome to our community! To help you get the best possible answer, I have tagged in our HIve experts @asish @smruti @tjangid @ggangadharan  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

the query is 

SELECT `iqrowkey`, `niu1`, `nif1`, `nombre1`, `niu2`, `nif2`, `nombre2`, `conyuge`, `exconyuge`, `ascendiente`, `descendiente`, `xd003`, `xfsinergi`, `xd001`, `xd0011` FROM `rstg_pro`.`rstg`

[#4]

hi @denysobukhov  Is your cluster SSL and LDAP enabled?

Are you able to connect from beeline?

Please review https://community.cloudera.com/t5/Community-Articles/How-to-Connect-to-Hiveserver2-Using-Cloudera-JD... and change it as per the usage.

[#5]

it's customers server and I don't have access to shell 

[#6]

@denysobukhov 

Confirm the JDBC version you are trying to connect? Try upgrading to the latest version https://www.cloudera.com/downloads/connectors/hive/jdbc/2-6-25.html 

If the above didn't help please attach the TRACE level driver logs. Refer https://docs.cloudera.com/documentation/other/connectors/hive-jdbc/2-6-25/Cloudera-JDBC-Connector-fo... page# 103.

[#7]

i'm using 

HiveJDBC42 2.6.25.1033

i'll try TRACE log level

[#8]

@denysobukhov 
Based on error stack trace which you have shared looks like the connection is getting closed before fetching the results for the query.
Is there a delay between submitting the query and then receiving the error? If yes, what is the delay exactly.
During the time of error appears,have you observe any alerts from HS2 ?
Are you able to  connect to HS2 from beeline without any issue during the time of  failure?
Is hive2host.loc is the loadbalancer URL?
We might require the JDBC TRACE LOGS and HS2 logs to determine the cause  of failure.

Thank You.

[#9]

I can't confirm that the issue is related to Hive/Client itself or their configuration. I received indirect confirmations that HMS were restarted during my client app was accessing it. 

[#10]

@denysobukhov 
If this issue hasn't been resolved I am suspecting the HS2 idle Timeout and Thread pool size. Can you please do the below and share the out come.
1. Address Server-Side Resource or Timeout Issues

Increase HiveServer2 Idle Timeout

By default, HiveServer2 may close idle connections after a certain period. Increase this timeout:

• Update the HiveServer2 config:
  • hive.server2.idle.session.timeout (default: 600000 ms / 10 minutes).
  • Set it to a larger value, e.g., 3600000 (1 hour).
• hive.server2.idle.operation.timeout (default: 5 minutes for operations).
• Increase to match your app's use case.
  Spoiler

  SET hive.server2.idle.session.timeout=3600000;
  SET hive.server2.idle.operation.timeout=3600000;

  Adjust Thread Pool Size

  • If HiveServer2 runs out of threads to handle requests, it can drop connections:
    • Increase hive.server2.threads to a higher value in HiveServer2 configurations.
    • Restart HiveServer2 after changes.
  First check the default hive.server2.thrift.max.worker.threads 
  
  Spoiler

  jstack -l <HiveServere2_ProccessId> | grep  ".Thread.Stat" | wc -l

  Happy hadooping