Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

CDH6.1.1 hiveserver2 thrift error

avatar
New Contributor

Hello,

 

         

hiveserver2 thrift error:

 

ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-40]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
... 4 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_131]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_131]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_131]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.1.1.jar:2.1.1-cdh6.1.1]

 

 

Please help.

 

Thanks,

ALAN

 

4 REPLIES 4

avatar
Super Guru
This could be just networking issue.

Was your HMS working at the time?
How often this happens? Is it intermittent or always? Or only when running certain queries?

Some context on the error might help.

Cheers

avatar
Expert Contributor
it happends in my env, it's very frequency, near one time/sec(it seems more faster). but hive service is normal. i have find some articles in google, some peoples mean if there is a load banlance in front, the LB will test the connection by tcp check, then the errors happend. is this true ?

avatar
Expert Contributor
i have tested manually by telnet, the below is the Error logs, so i think these errors actually are happend cause TCP check, how to filter these logs? 2019-07-18 02:34:44,783 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-86]: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -1 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181] at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.apache.thrift.transport.TTransportException: Invalid status -1 at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0]

avatar
Super Guru
Hi iamfromsky,

To confirm, are you able to change your LB's setting to check less frequent to see if the number of error/warning will also reduce?

However, based on my experience, the error produced by LB's tcp check normally is SASL related, not like "java.net.SocketException: Connection reset".

It would be good to modify LB setting to confirm though.

Cheers
Eric