Thank you for this very useful answer. Some remarks: 1)I have the feeling the row counter does help. When I let it run, the read time for that table afterwards is reduced by a factor of 3. I did not look into the details and would expect it just counts the rows and does not access the data, but my test suggests it does also store some of the data. 2) Isn't there any mapreduce job availalbe that will access all the data 1x then? If this were sql, I would write a order by or SORT(something that kicks off a full table scan)? I was also trying CellCounter, but this makes my regionservers become overloaded. 2) I have set the IN_MEMORY =true already for my tables. 3)For my use case I need to do lots of heavy aggregation queries. It is always on the same couple tables. I would prefer to have these cached completely. It would be very useful if there is a method to permanently cache them. I guess the best thing I can do is the in_memory=true, but it seems like it doesnt permanently store it. I have the feeling that after a while the cache is cleaned randomly. I guess this is what you mentioned about the LRU cache that will do this... I think I am just using the standard implementation (only changed it in cloudera manager using the hfile.block.cache.size to a higher value of 0.55). I did not understand this comment "We can use RegionServer Grouping for the concerned Table's Region & ensure the BlockCache of the concerned RegionServer Group is used for Selective Tables' Regions, thereby reducing the impact of LRU. ". So, there is a way to prevent that the cache is cleaned again for these tables? I can tell from the hbase master that the on-heap memory is only partially used (still more than 50% available, after I have accessed all my tables once). I have 180GB avaliable and the tables would fit in there. Too bad I can not permanently store them in memory. Relevant settings are: HBase: hbase.bucketsize.cache = 10 GB HBase: hfile.block.cache.size = 0.55 HBase: Java Heap Size HBASE regionserver = 20GB and at the catalog level: CACHE_DATA_ON_WRITE true CACHE_INDEX_ON_WRITE true PREFETCH_BLOCKS_ON_OPEN true IN_MEMORY true BLOOMFILTER ROWCOL 5)The article is very interesting-I am still digesting it. One important thing I see is that the cached data is decompressed (I was not aware of this). I see that it is recommended to have block sizes comparable to the row size of the data blocks you are interested in. Otherwise you are caching a majority of data you are not interested in. But I only use this cluster to access the same few tables which I all would like to have completely cached, so for me all data is relevant and can be cached, so maybe this will not help me. Finally, I should figure out how to know what the size is of 1 row of data. Once again, your responses are very advanced and helpful so thank you!
... View more
