Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎06-23-2017

Impala query plan time very long and thrift errors

Hello Everyone,

 

We are using Cloudera 5.8.3 and Impala 2.6.

 

We have an small java app that connects to impala using JDBC, and for warm-up it tries to read all the table schemas (using describes $TABLE).

 

Trying to speed up this process we have seen that it takes about 2 min in average to run each describe which is a really long time for what we need.

 

Having a look to the stats on cloudera manager we have seen this time distribution on query planning wait time.

Pasted image at 2017_06_23 11_34 AM.png

 

It seems that Impala is having timeouts connecting to somwhere and after that it does retries, but i have been looking for timeouts around 100 / 120 secs in the config and i haven't seen anything that matches that possible behaviour.

 

Also having a look to the logs we have seen many thrift time out exceptions like this.

Time    Log Level   Source  Log Message
Jun 23, 4:00:08.326 AM  ERROR   org.apache.thrift.server.TThreadPoolServer  
[HiveServer2-Handler-Pool: Thread-40]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
    at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
    at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
    at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
    at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
    ... 4 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
    ... 10 more

I suppose those things are somehow related, but i cannot see anything related.

 

Any idea of what can be happening?
Thanks a lot,

Adrian

Announcements