I am wondering what difference in IO can be expected for hbase with storage in the cloud VS storage on hdfs. I would expect that when data is retrieved from hdfs, it will be a lot faster than from the cloud (like adls-in my specific case adls gen2=abfs). Is there somewhere where I can test this? Or find a previous study for this? If this is the case, then one would expect that current hbase performance for reading data is a bit less than some years ago when everything was on premise and typically hdfs was used?     Maybe I am missing something obvious here, so any insight is appreciated !   ... View more

I have a question about regionservers that go to bad health when I am writing data from Spark. My question is: how do I know (in what logs to look exactly) what the cause is of the bad health? BACKGROUND: My spark job processes a month of some data and writes to hbase. This runs 1/month. For most months there are no problems. For some months (probably with slightly higher traffic), I notice the regionservers go into bad health. The master notices this and when the server goes down, it moves all regions to another regionserver, and then it becomes healthy again. But as my writing is going on, the same happens to other regionservers and eventually my spark job fails. I am confident my error in write is not due to corrupt data (like impossible UTC time or so), since sometimes this happens and then I clearly see in my spark logs "caused by value out of bounds". Now I see 21/03/17 15:54:22 INFO client.AsyncRequestFutureImpl: id=5, table=sometable, attempt=10/11, failureCount=2048ops, last exception=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server myregionserver4,16020,1615996455401 is not running yet LOGS: In the logs of the master I mainly see that it starts to identify all regions on the regionserver that is dying, and moves them. In the logs of the regionserver around the time of failure I noticed a "org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.39 GB, freeSize=2.07 GB, max=5.46 GB, blockCount=27792, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=1049, evicted=0, evictedPerRun=0.0 2021-03-17 15:52:31,442 INFO " but this might be not relevant. This is the behavior of the #of regions for the regionserver that dies: it dies here 2x and the number of regions collapses as they are moved. 2 failures of regionserver at 3.54 and 4.15, followed by copying of regions The cpu looks as follows (looks ok to me): The memory looks like it comes close to the max available:     --> Maybe it fails because of memory? Like that all writes are first written to memory and the available memory has been used up? But I had expected to see this kind of error message somewhere (logs or cloudera manager). These are the hbase settings (on CDP 7.2, hbase 2.2) 32 GB regionservers Java Heap Size HBASE regionserver =20GB hfile.block.cache.size =0.55 hbase.regionserver. memstore. global.size =0.25 hbase.bucketsize.cache=16GB   I am wondering how I can understand better what the reason is that the regionserver fails. Then I can change the hbase config accordingly. ... View more

We are trying to use hbase with adls as a storage layer. Is this still possible in cdh 6.1? This is described in the docs for cdh 5.12 : https://docs.cloudera.com/documentation/enterprise/5-12-x/topics/admin_using_adls_storage_with_hbase.html#hbase_adls_configuration_steps but we are using cdh 6.1 and there I don't see a similar page anymore.    Off course I ask since I have an issue following the 5.12 docs. When I want to change in Cloudera Manager in the hbase configuration,    hbase.root.dir = adl://ourproject.azuredatalakestore.net/hbase   I can not save this method as I get an error saying the variable is not in the allowed format:   HDFS Root Directory: Path adls://ourproject.azuredatalakestore.net/hbase does not conform to the pattern "(/[-+=_.a-zA-Z0-9]+)+(/)*".     ... View more