Reply
Highlighted
Explorer
Posts: 69
Registered: ‎01-24-2017
Accepted Solution

HBase: Thrift error occurred during processing of message

All of a sudden (I do not think it was really used), HBase shows red status. The log message says:

==============
Mar 6, 2:05:39.141 PM ERROR org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer
Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Bad version in readMessageBegin
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:223)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

=============

I have started HBase Thrift Server and so far it appears to be running.

 

The only recent significant change to the cluster I can think of was fully enabling TLS. Could that affect HBase?

 

Cloudera Employee
Posts: 25
Registered: ‎01-27-2017

Re: HBase: Thrift error occurred during processing of message

[ Edited ]

Igor, 

 

The log snippet you provided may be consistent with an error related to the heap size of the Thrift server. 

 

GoTo CM -----> HBase ----> Configuration -----> On the left side pane, under "Scope" Select HBaseThrift Server" ---> In the search box, search for "hbase.regionserver.thrift" ------> you should then see 2 properties with check boxes----->


Enable HBase Thrift Server Compact Protocol hbase.regionserver.thrift.compact

Enable HBase Thrift Server Framed Transport hbase.regionserver.thrift.framed


Check both the boxes -----> Save Changes

2) Increase the heap size of ThriftServer to 2GB. Save Changes.

3) CM => HBase => Instances => GoTo Instances page ---> Note that the Thrift Server would now show "roles started with outdated configuration" ----> Restart only the HBase Thrift roles.

From there I would monitor for a week and see if it is still crashing or not.

 

 

*for configuring HBase with TLS/SSL : 

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_ssl_hbase.html 

 

Thanks, 

Jordan 

 

Explorer
Posts: 69
Registered: ‎01-24-2017

Re: HBase: Thrift error occurred during processing of message

Hi Jordan,

Yes, Cloudera also recommended increasing the heap size and after I did it couple weeks ago, I did not see any more crashes. It is rather surprising though that the default configuration causes crashes. That raises the question how optimal or even acceptable other parameters are and how to tune them.

Thank you,

Igor

 

Announcements