Skip to main content
denysobukhov
denysobukhov
Silver ⭐️⭐️
September 4, 2024
Question

[Cloudera][HiveJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: Socket is closed by peer.

hi community cloudera, 

i'm experiencing strange hive drive behaviour:

my app consequently performing `select` queries on multiple tables successfully. But at some point it gets an error:

 

java.sql.SQLException: [Cloudera][HiveJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: Socket is closed by peer. FetchResults for query "SELECT `iq***".
 
and when the app retries to reconnect (because java.sql.Connection.isValid() == false)
it constantly gets 
 
[Cloudera][JDBC](10060) Connection has been closed.
java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10060) Connection has been closed.
at com.cloudera.hiveserver2.exceptions.ExceptionConverter.toSQLException(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.SConnection.closeConnection(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.SConnection.abortInternal(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.SConnection.close(Unknown Source)
 
Since connection string is 
jdbc:hive2://hive2host.loc:10000;AuthMech=3
 
I don't have access to Hive logs, so I can guess only.
Could it be relates to LDAP server issues? Or it is by some other reason? 
 
 
 

    9 replies

    VidyaSargur
    Community Manager
    VidyaSargur
    Community Manager
    September 5, 2024

    @denysobukhov, Welcome to our community! To help you get the best possible answer, I have tagged in our HIve experts @asish @smruti @tjangid @ggangadharan  who may be able to assist you further.

    Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

    Translate
    Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
      denysobukhov
      denysobukhovAuthor
      Silver ⭐️⭐️
      September 6, 2024

      the query is 

      SELECT `iqrowkey`, `niu1`, `nif1`, `nombre1`, `niu2`, `nif2`, `nombre2`, `conyuge`, `exconyuge`, `ascendiente`, `descendiente`, `xd003`, `xfsinergi`, `xd001`, `xd0011` FROM `rstg_pro`.`rstg`
      Translate
        asish
        Gold ⭐️⭐️⭐️
        asish
        Gold ⭐️⭐️⭐️
        September 5, 2024

        hi @denysobukhov  Is your cluster SSL and LDAP enabled?

        Are you able to connect from beeline?

        Please review https://community.cloudera.com/t5/Community-Articles/How-to-Connect-to-Hiveserver2-Using-Cloudera-JDBC-driver/ta-p/376336 and change it as per the usage.

        Translate
          denysobukhov
          denysobukhovAuthor
          Silver ⭐️⭐️
          September 9, 2024

          it's customers server and I don't have access to shell 

          Translate
          ajay_riddler
          Gold ⭐️
          ajay_riddler
          Gold ⭐️
          September 6, 2024

          @denysobukhov 

          Confirm the JDBC version you are trying to connect? Try upgrading to the latest version https://www.cloudera.com/downloads/connectors/hive/jdbc/2-6-25.html 

          If the above didn't help please attach the TRACE level driver logs. Refer https://docs.cloudera.com/documentation/other/connectors/hive-jdbc/2-6-25/Cloudera-JDBC-Connector-for-Apache-Hive-Install-Guide.pdf page# 103.

          Translate
          denysobukhov
          denysobukhovAuthor
          Silver ⭐️⭐️
          September 9, 2024

          i'm using 

          HiveJDBC42 2.6.25.1033

          i'll try TRACE log level

          Translate
          Mounika_Reddy
          Gold ⭐️
          Mounika_Reddy
          Gold ⭐️
          September 11, 2024

          @denysobukhov 
          Based on error stack trace which you have shared looks like the connection is getting closed before fetching the results for the query.
          Is there a delay between submitting the query and then receiving the error? If yes, what is the delay exactly.
          During the time of error appears,have you observe any alerts from HS2 ?
          Are you able to  connect to HS2 from beeline without any issue during the time of  failure?
          Is hive2host.loc is the loadbalancer URL?
          We might require the JDBC TRACE LOGS and HS2 logs to determine the cause  of failure.

          Thank You.

          Translate
            denysobukhov
            denysobukhovAuthor
            Silver ⭐️⭐️
            December 15, 2024

            I can't confirm that the issue is related to Hive/Client itself or their configuration. I received indirect confirmations that HMS were restarted during my client app was accessing it. 

            Translate
            Shelton
            Shelton
            Gold ⭐️⭐️⭐️
            December 17, 2024

            @denysobukhov 
            If this issue hasn't been resolved I am suspecting the HS2 idle Timeout and Thread pool size. Can you please do the below and share the out come.
            1. Address Server-Side Resource or Timeout Issues

            Increase HiveServer2 Idle Timeout

            By default, HiveServer2 may close idle connections after a certain period. Increase this timeout:

            • Update the HiveServer2 config:
              • hive.server2.idle.session.timeout (default: 600000 ms / 10 minutes).
              • Set it to a larger value, e.g., 3600000 (1 hour).
            • hive.server2.idle.operation.timeout (default: 5 minutes for operations).
            • Increase to match your app's use case.
              Spoiler
              SET hive.server2.idle.session.timeout=3600000;
              SET hive.server2.idle.operation.timeout=3600000;
              Adjust Thread Pool Size
              • If HiveServer2 runs out of threads to handle requests, it can drop connections:
                • Increase hive.server2.threads to a higher value in HiveServer2 configurations.
                • Restart HiveServer2 after changes.
              First check the default hive.server2.thrift.max.worker.threads 
              Spoiler

              jstack -l <HiveServere2_ProccessId> | grep ".Thread.Stat" | wc -l

              Happy hadooping


             

             

             

             

             

             

             

             

             

             

             

             

             

             

             

             

            Translate
            Terms & ConditionsAccessibility statement