HBase: Thrift error occurred during processing of message

[#1]

All of a sudden (I do not think it was really used), HBase shows red status. The log message says:

==============
Mar 6, 2:05:39.141 PM ERROR org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer
Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Bad version in readMessageBegin
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:223)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

=============

I have started HBase Thrift Server and so far it appears to be running.

The only recent significant change to the cluster I can think of was fully enabling TLS. Could that affect HBase?

[#2 · Accepted solution]

Igor, 

The log snippet you provided may be consistent with an error related to the heap size of the Thrift server. 

GoTo CM -----> HBase ----> Configuration -----> On the left side pane, under "Scope" Select HBaseThrift Server" ---> In the search box, search for "hbase.regionserver.thrift" ------> you should then see 2 properties with check boxes----->


Enable HBase Thrift Server Compact Protocol hbase.regionserver.thrift.compact

Enable HBase Thrift Server Framed Transport hbase.regionserver.thrift.framed


Check both the boxes -----> Save Changes

2) Increase the heap size of ThriftServer to 2GB. Save Changes.

3) CM => HBase => Instances => GoTo Instances page ---> Note that the Thrift Server would now show "roles started with outdated configuration" ----> Restart only the HBase Thrift roles.

From there I would monitor for a week and see if it is still crashing or not.

*for configuring HBase with TLS/SSL : 

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_ssl_hbase.html 

Thanks, 

Jordan 

[#3]

Hi Jordan,

Yes, Cloudera also recommended increasing the heap size and after I did it couple weeks ago, I did not see any more crashes. It is rather surprising though that the default configuration causes crashes. That raises the question how optimal or even acceptable other parameters are and how to tune them.

Thank you,

Igor