Skip to main content
JB0000000000001
JB0000000000001
Gold ⭐️
May 19, 2021
Solved

simplest method to read a full hbase table so it is stored in the cache

Dear Experts,
I am interested in accessing all the data in a hbase table once, so it will be cached.
I am using an application that reads from hbase and I notice it is a lot faster when the data was read before, so I would like to "preheat" the data by warming it up: accessing it all once, so it is stored in the cache.
I am quite familiar with the cluster configuration necessary to enable caching but wonder what is the easiest way to do this.
Is there a command that accesses all the data in all regions 1x?


 

smdas
smdasAnswer
Gold ⭐️⭐️
May 21, 2021

Hello @JB0000000000001 

 

Appreciate your detailed response. To some of your reviews,

  • To check upon the Impact of Row Counter on Caching, Created a Table on a Vanilla HBase Setup, Executed PE Write to insert 100,000 Rows. Flush & Restarted the Service to ensure there is no WAL Split write going through the MemStore, thereby assisting in Read Access. As shown below, there is no Read & Write with Server Metrics showing 4 Blocks of 400KB Size in Block Cache:

Before_RowCounter_1.pngBefore_RowCounter_2.png

  • After a RowCounter Job, We observed ~100,000 Reads with Used Heap increased around ~150MB yet Block Cache Size remained Similar.

After_RowCounter_1.pngAfter_RowCounter_2.png

  • After a Scan Operation reading each of the 100,000 Rows, the Read Request increased by 100,000 & the Block Cache Sizing increased by ~100MB & Block Count increasing by ~1500 & Hit Ration further deteriorating owing to larger Misses:

After_Scan_1.pngAfter_Scan_2.png

  • Row Counter likely doesn't offer Caching benefits yet I have shared our Observation to ensure you can compare the same with your observation as well.
  • Using the Region Server Grouping would still involve Eviction, yet the Competition & Quantity of Objects for Cache would be reduced by reducing the Tables involved within the RS Grouping Scope. This might be a bit Excessive approach yet I thought of sharing the same, in case you may wish to review.
  • The Link [1] has the required Map Reduce Programs & I couldn't find anyone, which may fit the Use-Case of access all Data 1x. A Full Table Scan can be likely equivalent to running a Scan on HBase Shell for the Table. Or, Phoenix for SQL approach may be considered by you yet I believe writing a SELECT SQL using FTS is equivalent to Scanning the HBase Table via "scan".
  • Link [2] does mention In-Memory Column Family are the last to evict, yet there's no guarantee that In-Memory Column Family would always remain in Memory.
  • Also, I see you have "CACHE_INDEX_ON_WRITE" as True. So, I assume you might be extracting the most from your Memory unless Caching Policies possibility. While I haven't tested, Sharing Link [3] which talks about the various Caching Policies Test for Block Cache.

While I couldn't find any 100% guaranteed way to Cache the Objects, I believe you have implemented the most such functionalities already as discussed until now. I fear Compression & Encoding may not be helpful as Data is always Decompressed in Memory, likely leaving the possibility of reviewing the Caching Policies. I may have missed other possibilities & would share, if I come across any.

 

- Smarak

 

[1] https://hbase.apache.org/book.html#hbase.mapreduce.classpath

[2] https://hbase.apache.org/book.html#block.cache.design

[3] https://blogs.apache.org/hbase/entry/comparing_blockcache_deploys

 

 

View in thread

8 replies

JB0000000000001
JB0000000000001Author
Gold ⭐️
May 19, 2021

I have the feeling this is a good guess:
hbase org.apache.hadoop.hbase.mapreduce.RowCounter
It counts rows, but not entirely sure it will cache everything.

Translate
smdas
Gold ⭐️⭐️
smdas
Gold ⭐️⭐️
May 21, 2021

Hello @JB0000000000001 

 

Thanks for using Cloudera Community. You wish to store a Table in Cache via a Read-Once Model. 

 

  • As you stated, RowCounter would count the Rows & skip caching the data in the Cache as well,
  • Even if you have a Block Cache configured with sufficient capacity to manage the Table's Data into the Cache, their LRU implementation would likely cause the Object to be evicted. I haven't used any other Cache implementation outside of LRU to comment. 
  • If you have a Set of Tables meeting such requirement, We can use RegionServer Grouping for the concerned Table's Region & ensure the BlockCache of the concerned RegionServer Group is used for Selective Tables' Regions, thereby reducing the impact of LRU. 
  • Test using the "IN_MEMORY" Flag for the Table's Column Families, which would try to persist the Column's Family Data as long as it can without any guarantee. 
  • While an Old Blog, yet [1] offers a few Practices implemented by a Heavy-HBase-Using Customer of Cloudera, written by their Employees' experience. As you are already familiar with various Heap Option, there may be no new Information for you yet I am sharing for closing any loop. 

Hope the above helps. Do let us know if your Team implemented any approach so as to benefit the wider audience looking at Similar Use-Case.

 

- Smarak

 

[1] http://blog.asquareb.com/blog/2014/11/21/leverage-hbase-cache-and-improve-read-performance/

Translate
    JB0000000000001
    JB0000000000001Author
    Gold ⭐️
    May 21, 2021

    Thank you for this very useful answer.

    Some remarks:

    1)I have the feeling the row counter does help. When I let it run, the read time for that table afterwards is reduced by a factor of 3. I did not look into the details and would expect it just counts the rows and does not access the data, but my test suggests it does also store some of the data.

    2) Isn't there any mapreduce job availalbe that will access all the data 1x then? If this were sql, I would write a order by or SORT(something that kicks off a full table scan)? I was also trying CellCounter, but this makes my regionservers become overloaded.

    2) I have set the IN_MEMORY =true already for my tables.

    3)For my use case I need to do lots of heavy aggregation queries. It is always on the same couple tables. I would prefer to have these cached completely. It would be very useful if there is a method to permanently cache them. I guess the best thing I can do is the in_memory=true, but it seems like it doesnt permanently store it. I have the feeling that after a while the cache is cleaned randomly. I guess this is what you mentioned about the LRU cache that will do this... I think I am just using the standard implementation (only changed it in cloudera manager using the hfile.block.cache.size to a higher value of 0.55). 
    I did not understand this comment
    "We can use RegionServer Grouping for the concerned Table's Region & ensure the BlockCache of the concerned RegionServer Group is used for Selective Tables' Regions, thereby reducing the impact of LRU. ". So, there is a way to prevent that the cache is cleaned again for these tables?

    I can tell from the hbase master that the on-heap memory is only partially used (still more than 50% available, after I have accessed all my tables once). I have 180GB avaliable and the tables would fit in there. Too bad I can not permanently store them in memory.

    Relevant settings are:

    HBase: hbase.bucketsize.cache = 10 GB

    HBase: hfile.block.cache.size = 0.55 

    HBase: Java Heap Size HBASE regionserver = 20GB

    and at the catalog level:

    CACHE_DATA_ON_WRITE

    true

    CACHE_INDEX_ON_WRITE

    true

    PREFETCH_BLOCKS_ON_OPEN

    true

    IN_MEMORY

    true

    BLOOMFILTER

    ROWCOL

     

    5)The article is very interesting-I am still digesting it. One important thing I see is that the cached data is decompressed (I was not aware of this). 
    I see that it is recommended to have block sizes comparable to the row size of the data blocks you are interested in. Otherwise you are caching a majority of data you are not interested in. But I only use this cluster to access the same few tables which I all would like to have completely cached, so for me all data is relevant and can be cached, so maybe this will not help me.

    Finally, I should figure out how to know what the size is of 1 row of data.

    Once again, your responses are very advanced and helpful so thank you!

     

     

     

    Translate
    Terms & ConditionsAccessibility statement