Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase: Thrift error occurred during processing of message

Solved Go to solution
Highlighted

HBase: Thrift error occurred during processing of message

All of a sudden (I do not think it was really used), HBase shows red status. The log message says:

==============
Mar 6, 2:05:39.141 PM ERROR org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer
Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Bad version in readMessageBegin
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:223)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

=============

I have started HBase Thrift Server and so far it appears to be running.

 

The only recent significant change to the cluster I can think of was fully enabling TLS. Could that affect HBase?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HBase: Thrift error occurred during processing of message

Contributor

Igor, 

 

The log snippet you provided may be consistent with an error related to the heap size of the Thrift server. 

 

GoTo CM -----> HBase ----> Configuration -----> On the left side pane, under "Scope" Select HBaseThrift Server" ---> In the search box, search for "hbase.regionserver.thrift" ------> you should then see 2 properties with check boxes----->


Enable HBase Thrift Server Compact Protocol hbase.regionserver.thrift.compact

Enable HBase Thrift Server Framed Transport hbase.regionserver.thrift.framed


Check both the boxes -----> Save Changes

2) Increase the heap size of ThriftServer to 2GB. Save Changes.

3) CM => HBase => Instances => GoTo Instances page ---> Note that the Thrift Server would now show "roles started with outdated configuration" ----> Restart only the HBase Thrift roles.

From there I would monitor for a week and see if it is still crashing or not.

 

 

*for configuring HBase with TLS/SSL : 

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_ssl_hbase.html 

 

Thanks, 

Jordan 

 

2 REPLIES 2

Re: HBase: Thrift error occurred during processing of message

Contributor

Igor, 

 

The log snippet you provided may be consistent with an error related to the heap size of the Thrift server. 

 

GoTo CM -----> HBase ----> Configuration -----> On the left side pane, under "Scope" Select HBaseThrift Server" ---> In the search box, search for "hbase.regionserver.thrift" ------> you should then see 2 properties with check boxes----->


Enable HBase Thrift Server Compact Protocol hbase.regionserver.thrift.compact

Enable HBase Thrift Server Framed Transport hbase.regionserver.thrift.framed


Check both the boxes -----> Save Changes

2) Increase the heap size of ThriftServer to 2GB. Save Changes.

3) CM => HBase => Instances => GoTo Instances page ---> Note that the Thrift Server would now show "roles started with outdated configuration" ----> Restart only the HBase Thrift roles.

From there I would monitor for a week and see if it is still crashing or not.

 

 

*for configuring HBase with TLS/SSL : 

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_ssl_hbase.html 

 

Thanks, 

Jordan 

 

Re: HBase: Thrift error occurred during processing of message

Hi Jordan,

Yes, Cloudera also recommended increasing the heap size and after I did it couple weeks ago, I did not see any more crashes. It is rather surprising though that the default configuration causes crashes. That raises the question how optimal or even acceptable other parameters are and how to tune them.

Thank you,

Igor