CDH6.1.1 hiveserver2 thrift error

[#1]

Hello,

hiveserver2 thrift error:

ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-40]: Error occurred during processing of message.

java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset

at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset

at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

... 4 more

Caused by: java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_131]

at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131]

at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_131]

at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_131]

at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_131]

at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

Please help.

Thanks,

ALAN

[#2]

This could be just networking issue.

Was your HMS working at the time?
How often this happens? Is it intermittent or always? Or only when running certain queries?

Some context on the error might help.

Cheers

[#3]

it happends in my env, it's very frequency, near one time/sec(it seems more faster). but hive service is normal. i have find some articles in google, some peoples mean if there is a load banlance in front, the LB will test the connection by tcp check, then the errors happend. is this true ?

[#4]

i have tested manually by telnet, the below is the Error logs, so i think these errors actually are happend cause TCP check, how to filter these logs? 2019-07-18 02:34:44,783 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-86]: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -1 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181] at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.apache.thrift.transport.TTransportException: Invalid status -1 at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0]

[#5]

Hi iamfromsky,

To confirm, are you able to change your LB's setting to check less frequent to see if the number of error/warning will also reduce?

However, based on my experience, the error produced by LB's tcp check normally is SASL related, not like "java.net.SocketException: Connection reset".

It would be good to modify LB setting to confirm though.

Cheers
Eric