question Re: HBase: Thrift error occurred during processing of message in Archives of Support Questions (Read Only)

HBase: Thrift error occurred during processing of message

IgorYakushin — Fri, 16 Sep 2022 11:12:42 GMT

All of a sudden (I do not think it was really used), HBase shows red status. The log message says:

==============
Mar 6, 2:05:39.141 PM ERROR org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer
Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Bad version in readMessageBegin
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:223)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

=============

I have started HBase Thrift Server and so far it appears to be running.

The only recent significant change to the cluster I can think of was fully enabling TLS. Could that affect HBase?

Re: HBase: Thrift error occurred during processing of message

Borg — Thu, 06 Apr 2017 21:21:14 GMT

Igor,

The log snippet you provided may be consistent with an error related to the heap size of the Thrift server.

GoTo CM -----> HBase ----> Configuration -----> On the left side pane, under "Scope" Select HBaseThrift Server" ---> In the search box, search for "hbase.regionserver.thrift" ------> you should then see 2 properties with check boxes----->

Enable HBase Thrift Server Compact Protocol hbase.regionserver.thrift.compact

Enable HBase Thrift Server Framed Transport hbase.regionserver.thrift.framed

Check both the boxes -----> Save Changes

2) Increase the heap size of ThriftServer to 2GB. Save Changes.

3) CM => HBase => Instances => GoTo Instances page ---> Note that the Thrift Server would now show "roles started with outdated configuration" ----> Restart only the HBase Thrift roles.

From there I would monitor for a week and see if it is still crashing or not.

*for configuring HBase with TLS/SSL :

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_ssl_hbase.html

Thanks,

Jordan

Re: HBase: Thrift error occurred during processing of message

IgorYakushin — Fri, 07 Apr 2017 04:36:24 GMT

Hi Jordan,

Yes, Cloudera also recommended increasing the heap size and after I did it couple weeks ago, I did not see any more crashes. It is rather surprising though that the default configuration causes crashes. That raises the question how optimal or even acceptable other parameters are and how to tune them.

Thank you,

Igor