Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issues with storm-hbase connectors! Issue with either of "HBaseLookupBolt" or "HBaseBolt" or HBase Table update!!

Issues with storm-hbase connectors! Issue with either of "HBaseLookupBolt" or "HBaseBolt" or HBase Table update!!

Rising Star

I have storm topology, which reads messages from kafka, does a lookup from hbase table, apply some business login on message and update hbase table!c

Below are steps, how a message flow from kafka to hbase happens in my storm topolgy!!
  1. Read message from kafka spout (Kafka Spout)
  2. Extract id column from the message read from kafka in step1 (General Bolt)
  3. Do a lookup from hbase table for the id ( storm-hbase connector – HbaseLookup Bolt)
  4. Based on the retrieved value from the Hbase in step3, apply business logic, then update the same HBase table used for lookup, for the id.

Issue Storm hbase lookup bolt is reading wrong value from the HBase table!

To explain in more detail:

when incoming messages arrive at a time gap of few milli secs to secs, "HBaseLookupBolt" is reading old value from HBase table, even though the table is updated already using "HBaseBolt" !!!

Can someone please share your thoughts, on how can i ensure lookup bolt takes the right value ?

Or is it possible thru any setting, to slow down the speed at which storm processes the records or any other of your valuable thoughts.... ?

3 REPLIES 3
Highlighted

Re: Issues with storm-hbase connectors! Issue with either of "HBaseLookupBolt" or "HBaseBolt" or HBase Table update!!

Rising Star

Hi @Raja A , Are you using native api or phoenix to update records in HBase. For native API are you flushing commits after every update? (in case of phoenix are you commiting updates?)

Re: Issues with storm-hbase connectors! Issue with either of "HBaseLookupBolt" or "HBaseBolt" or HBase Table update!!

Rising Star

hi @Ajay, i am using storm-hbase connector api, which comes with Storm version 0.10.0.2.3.4.21-2.

And i have checked the source code of it, it does update for every record it processes!!

I am NOT using Phoenix.

Re: Issues with storm-hbase connectors! Issue with either of "HBaseLookupBolt" or "HBaseBolt" or HBase Table update!!

Rising Star

@Raja A There are couple of open libs for Storm-Hbase connector. Which one you are using?

If you are using HBaseBolt with TupleTableConfig class,try setting batching to false. (i.e setBatch(false)) By default it is true.

Don't have an account?
Coming from Hortonworks? Activate your account here