hbase performance with storage in the cloud versus on hdfs:

Question

I am wondering what difference in IO can be expected for hbase with storage in the cloud VS storage on hdfs.
I would expect that when data is retrieved from hdfs, it will be a lot faster than from the cloud (like adls-in my specific case adls gen2=abfs).

Is there somewhere where I can test this?

Or find a previous study for this?

If this is the case, then one would expect that current hbase performance for reading data is a bit less than some years ago when everything was on premise and typically hdfs was used?

Maybe I am missing something obvious here, so any insight is appreciated !

smdas · Accepted Answer

Hello @JB0000000000001

Thanks for using Cloudera Community. This is an Old Post, as such I am unsure whether you have found the details being shared below.

Having said that, [1] by Cloudera HBase Team offers a few details of S3 Performance in the final paragrapgh. As mentioned in the concerned Paragraph, BucketCache is Critical for Performance & AWS suggest the same in [2] under "Operational considerations" Section.

In short, there are definitely areas for Performance Degradation yet there are few Links is LinkedIn Slideshare around Microsoft, Airbnb, Huawei HBase Usage on Cloud & they offer a lot of details into the Observation made & Optimisation performed. The Optimisation aren't always centered on HBase & pivot towards Cloud Infrastructure as well. Such Slideshare should offer additional insight to your Team.

Being an Old Post, We wish to check if your Team have come across findings, which can be shared with us. This would allow the Post to be used by fellow community Users, who may be considering the Cloud Storage Latency aspect as well.

- Smarak

[1] https://blog.cloudera.com/how-hbase-in-cdp-can-leverage-amazons-s3/

[2] https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hbase-s3.html

smdas · Answer

Hello @JB0000000000001

We wish to follow-up with you on the Post & confirm if you have any additional Observation to be shared with respect to studying or implementing HBase on Cloud Storage. Or, our response to your Post was helpful in getting the required Cloud Storage possible latencies.

- Smarak

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.