Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

When is HBase CF TTL too short?

avatar
Rising Star

Would a short CF TTL of 30 minutes on 2 to 36 million row table be detrimental to performance?

This HBase table would be queried or inserted from Storm at a goal rate of 20k entries per second at peak performance and would like the current entries to expire after 30 minutes.

1 ACCEPTED SOLUTION

avatar
Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Master Mentor

@Enis @Devaraj Das I'd be curious to know the answer, my guess would be no, how many records inserted a day?

avatar

For the sake of discussion, lets say

  • the system is running at peak load 24 hours..
  • Out of 20K, there are 10K reads and 10K inserts per second

So, after the first 30 mins of running, system will add additional 10K deletes per second so a total of 30K hits. Its definitely not that straight forward and HBase is going to batch the actual deletes somehow internally.

30K tps is not a lot for HBase but the question is how big of a cluster are we talking about?

Also, other thing will be the memory available to the RegionServer... it makes sense to keep as much data in memory as possible so the I/O is minimal, as the data is to be deleted after 30 mins anyways. So, the next set of questions is - whats the memory available on the box and to region server? How big is each message?

avatar
Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Rising Star

Thanks Enis!