Hello @william266455    Thanks for using Cloudera Community. Your Post covers an extremely wide domain yet I would like to share my perspective from working with Customers on CDP Public Cloud: From the 3 Cloud Vendors, AWS was the Pioneer to start with & as such, I wish to point to the Link [1], which offers the 6 Major Advantages of Cloud Computing from an AWS WhitePaper.  Listing them here, We have "Trading Capital Expenses With Operational Expenses", "Scale", "Stop Guessing Capacity", "Agility", "Go Global", "Avoid Running & Maintaining Data Centre". The PDF does an awesome job of explaining the Benefits & I would skip reiterating the same.    As you posted the Question in Cloudera Community, I would like to take a few minutes sharing the Current On-Premise Model Short-Coming from Hadoop Ecosystem perspective (Note that On-Premise has it's Advantages & Stating On-Premise is Defunct would be an Over-Statement): Associated "Big Data" with "Hadoop" isn't new (Although the Focus should be moving from Data Size/Type to Analytics & Value from Data). Hadoop Ecosystem offers a wide variety of Services & they fit Use-Cases with different needs. As the Services increase, So does the Data being processed & reviewed. As such, any Capacity Planning to hard to begin with.  Be it Scaling Up or Scaling Out, Managing Linux or Windows Server isn't Easy. A Simple Task of adding Resources to an Instance isn't performed without Operational Experience & adding New Instances requires getting in touch with Multiple Parties (Admin, Billing, Vendor, Transport, Sponsor etc) Other context like Agility, Monitoring, Scaling, Routing are Self Explanatory.     Now, How does Cloudera helps via Cloudera Data Platform (CDP) is as follows: By ensuring Users only bother about their Use-Case (Running a Spark Job, Hive-Impala SQL, NoSQL HBase Analytics etc.) like the Users have been doing on their On-Premise, yet with the Flexibility of Cloud. The Best Part is a Customer can have 1 Setup on AWS, another Setup on Azure or GCP as well, yet the UI to perform your Hive/Spark/Kafka/HBase Jobs remains Similar.  When no Jobs are running, the Scale-Down happens implicitly & when Jobs are running, the Scale-Up happens without any User's intervention. This ensure User focus on their Job only & Cost is Saved as well.  Avoid Noisy Neighbours by ensuring the Jobs runs via Containers with their own resources & avoid any poor Job take away Host level resources, thereby affecting other Jobs.    As you tag "Cloudera Data Engineering", I wish to share a bit of details on the same. CDE allows you to run Spark Job (Just like you run on On-Premise) yet with the Flexibility of running them on Containers as well. And the Best Part is, End-User don't have to know anything about the Kubernetes Cluster running their Spark Job.    In short, Cloud offers Flexibility, Agility which is hard to achieve with On-Premise Setup. Yet, with Power comes Responsibility & here, AWS have a Shared-Responsibility Model as shared via [2]. Finally, I wish to share Cloudera Public Cloud Offering via [3] & some Awesome Free Training Videos on CDP via [4].    The above Opinion is Completely mine from my Experiences & I have barely scratched the Surface of the benefits from Cloud. I am happy to answer any specific query you may have from the above details. Also, Note that each of the AWS Link or PDF are Public Materials.    - Smarak   [1] https://docs.aws.amazon.com/whitepapers/latest/aws-overview/aws-overview.pdf#six-advantages-of-cloud-computing [2] https://aws.amazon.com/compliance/shared-responsibility-model/?ref=wellarchitected [3] https://docs.cloudera.com/cdp/latest/index.html [4] https://www.cloudera.com/about/training.html#?fq=training%3Acomplimentary%2Ffree ... View more

Hello @Priyanka26    As we haven't heard from your side, We shall summarise the Discussion in the Post to ensure the same benefits Users with similar experiences:   PROBLEM: In HDP v3.1.0, HBase NameSpace Region isn't assigned, thereby causing the following Message: 2021-03-17 20:29:54,614 WARN [Thread-18] master.HMaster: hbase:namespace,,1575575842296.0c72d4be7e562a2ec8a86c3ec830bdc5. is NOT online; state={0c72d4be7e562a2ec8a86c3ec830bdc5 state=OPEN, ts=1616010947554, server=itk-phx-prod-compute-6.datalake.phx,16020,1615483461273}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.  Your Team tried to use HBCK2 Assign yet the same fails with the following Error: Caused by: org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to itk-phx-prod-master-2.datalake.phx/192.168.15.180:16000 failed on local exception: java.io.IOException: Failed to specify server's Kerberos principal name   DISCUSSION SUMMARY: (I) In Customer's HDP v3.1.0, We have a Bug wherein the HBCK2 JAR can't used with the available Hbase-Client & Hbase-Server JAR in a Secure Cluster. There is no issue with the way your team is using the HBCK2. Owing to the Bug being mentioned above, the HBCK2 Jar is throwing the concerned exception. Without the modified Hbase-Client & Hbase-Server JAR, We can try to re-initialize the HBase Cluster yet only if the same isn't a Production Cluster.  (II) The referred JAR aren't available for download publicly. Unfortunately, I am not familiar with any other means other than manual intervention (Start HBase on a new DataDir & Bulkload from previous DataDir being one of them). Such issues aren't present in HDP v3.1.5 onwards.  (III) Your Team decided to use the Bulk-Load approach to ensure HBase is Initialised afresh. [1] shares the Steps used by your Team.    In short, Do Upgrade to HDP v3.1.5 (The same would be a Maintenance Upgrade from v3.1.0 to v3.1.5) as soon as possible. Until then, Such issues require Bulk-Loading. The Bug causing the HBCK2 issue in a Kerberized Environment impacts HDP v3.0.0 through (And inclusive) HDP v3.1.4 & Fixed in HDP v3.1.5.    Thanks again for using Cloudera Community.   - Smarak   [1] https://community.cloudera.com/t5/Support-Questions/Hbase-namespace-table-in-not-online/m-p/313460/highlight/true#M225541  ... View more

Hello @JB0000000000001    Appreciate your detailed response. To some of your reviews, To check upon the Impact of Row Counter on Caching, Created a Table on a Vanilla HBase Setup, Executed PE Write to insert 100,000 Rows. Flush & Restarted the Service to ensure there is no WAL Split write going through the MemStore, thereby assisting in Read Access. As shown below, there is no Read & Write with Server Metrics showing 4 Blocks of 400KB Size in Block Cache: After a RowCounter Job, We observed ~100,000 Reads with Used Heap increased around ~150MB yet Block Cache Size remained Similar. After a Scan Operation reading each of the 100,000 Rows, the Read Request increased by 100,000 & the Block Cache Sizing increased by ~100MB & Block Count increasing by ~1500 & Hit Ration further deteriorating owing to larger Misses: Row Counter likely doesn't offer Caching benefits yet I have shared our Observation to ensure you can compare the same with your observation as well. Using the Region Server Grouping would still involve Eviction, yet the Competition & Quantity of Objects for Cache would be reduced by reducing the Tables involved within the RS Grouping Scope. This might be a bit Excessive approach yet I thought of sharing the same, in case you may wish to review. The Link [1] has the required Map Reduce Programs & I couldn't find anyone, which may fit the Use-Case of access all Data 1x. A Full Table Scan can be likely equivalent to running a Scan on HBase Shell for the Table. Or, Phoenix for SQL approach may be considered by you yet I believe writing a SELECT SQL using FTS is equivalent to Scanning the HBase Table via "scan". Link [2] does mention In-Memory Column Family are the last to evict, yet there's no guarantee that In-Memory Column Family would always remain in Memory. Also, I see you have "CACHE_INDEX_ON_WRITE" as True. So, I assume you might be extracting the most from your Memory unless Caching Policies possibility. While I haven't tested, Sharing Link [3] which talks about the various Caching Policies Test for Block Cache. While I couldn't find any 100% guaranteed way to Cache the Objects, I believe you have implemented the most such functionalities already as discussed until now. I fear Compression & Encoding may not be helpful as Data is always Decompressed in Memory, likely leaving the possibility of reviewing the Caching Policies. I may have missed other possibilities & would share, if I come across any.   - Smarak   [1] https://hbase.apache.org/book.html#hbase.mapreduce.classpath [2] https://hbase.apache.org/book.html#block.cache.design [3] https://blogs.apache.org/hbase/entry/comparing_blockcache_deploys     ... View more