Created on 03-02-2019 04:53 PM - edited 09-16-2022 07:12 AM
Hello.
I am trying to set up a single-node cluster running Impala on Kudu with the HiveMetaStore being managed by PostgreSQL. Connecting to the database fails with a "java.net.SocketTimeoutException: Read timed out" when attempting to read from the Hive metastore through the com.cloudera.impala.jdbc41.Driver.
I have attempted to troubleshoot the installation using only the PostgreSQL driver, which connects correctly to the database and runs the intended workload without issues. Similarly, the tables residing on Kudu can be queried without issues from the impala-shell.
The installation is based on the CDH6.x-snapshot with hive-2.1.1 and the full stacktrace can be found below.
Let me know if there are other configuration options that may be of interest.
java.lang.RuntimeException: Failed to connect to database at pt.haslab.htapbench.api.Worker.<init>(Worker.java:89) at pt.haslab.htapbench.benchmark.TPCCWorker.<init>(TPCCWorker.java:72) at pt.haslab.htapbench.benchmark.HTAPBenchmark.createTerminals(HTAPBenchmark.java:189) at pt.haslab.htapbench.benchmark.HTAPBenchmark.makeWorkersImpl(HTAPBenchmark.java:88) at pt.haslab.htapbench.api.BenchmarkModule.makeWorkers(BenchmarkModule.java:209) at pt.haslab.htapbench.core.HTAPBench.runHybridWorkload(HTAPBench.java:846) at pt.haslab.htapbench.core.HTAPBench.main(HTAPBench.java:666) Caused by: java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500151) Error setting/closing session: java.net.SocketTimeoutException: Read timed out. at com.cloudera.impala.hivecommon.api.HS2Client.openSession(Unknown Source) at com.cloudera.impala.hivecommon.api.HS2Client.<init>(Unknown Source) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source) at com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) at com.cloudera.impala.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source) at com.cloudera.impala.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.cloudera.impala.jdbc.common.AbstractDriver.connect(Unknown Source) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at pt.haslab.htapbench.api.BenchmarkModule.makeConnection(BenchmarkModule.java:106) at pt.haslab.htapbench.api.Worker.<init>(Worker.java:85) at pt.haslab.htapbench.benchmark.TPCCWorker.<init>(TPCCWorker.java:72) at pt.haslab.htapbench.benchmark.HTAPBenchmark.createTerminals(HTAPBenchmark.java:189) at pt.haslab.htapbench.benchmark.HTAPBenchmark.makeWorkersImpl(HTAPBenchmark.java:88) at pt.haslab.htapbench.api.BenchmarkModule.makeWorkers(BenchmarkModule.java:209) at pt.haslab.htapbench.core.HTAPBench.runHybridWorkload(HTAPBench.java:846) Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Cloudera][ImpalaJDBCDriver](500151) Error setting/closing session: java.net.SocketTimeoutException: Read timed out. ... 17 more Caused by: com.cloudera.impala.jdbc41.internal.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at com.cloudera.impala.jdbc41.internal.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at com.cloudera.impala.jdbc41.internal.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at com.cloudera.impala.jdbc41.internal.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at com.cloudera.impala.jdbc41.internal.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at com.cloudera.impala.jdbc41.internal.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at com.cloudera.impala.jdbc41.internal.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at com.cloudera.impala.jdbc41.internal.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160) at com.cloudera.impala.hivecommon.api.HS2ClientWrapper.recv_OpenSession(Unknown Source) at com.cloudera.impala.jdbc41.internal.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147) at com.cloudera.impala.hivecommon.api.HS2ClientWrapper.OpenSession(Unknown Source) at com.cloudera.impala.hivecommon.api.HS2Client.openSession(Unknown Source) at com.cloudera.impala.hivecommon.api.HS2Client.<init>(Unknown Source) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source) at com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) at com.cloudera.impala.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source) at com.cloudera.impala.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.cloudera.impala.jdbc.common.AbstractDriver.connect(Unknown Source) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at pt.haslab.htapbench.api.BenchmarkModule.makeConnection(BenchmarkModule.java:106) at pt.haslab.htapbench.api.Worker.<init>(Worker.java:85) at pt.haslab.htapbench.benchmark.TPCCWorker.<init>(TPCCWorker.java:72) at pt.haslab.htapbench.benchmark.HTAPBenchmark.createTerminals(HTAPBenchmark.java:189) at pt.haslab.htapbench.benchmark.HTAPBenchmark.makeWorkersImpl(HTAPBenchmark.java:88) at pt.haslab.htapbench.api.BenchmarkModule.makeWorkers(BenchmarkModule.java:209) at pt.haslab.htapbench.core.HTAPBench.runHybridWorkload(HTAPBench.java:846) at pt.haslab.htapbench.core.HTAPBench.main(HTAPBench.java:666) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at com.cloudera.impala.jdbc41.internal.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 27 more
Created 03-05-2019 04:38 AM
The connection actually worked as intended, but I spawned more than the default number of max connections, which caused the later connections to time out. I solved the issue by starting the Impala cluster with the command:
bin/start-impala-cluster.py --impalad_args="--fe_service_threads 128"