Hello @Ellengogo    I am not able to find the exact Case reference as this Post is few months old. Typically, such issues are caused by Resource Constraint, wherein the Engine Pod (Created when a Workbench Session is started) gets terminated owing to Resource Constraint. If the issue is persistent, a Support Case would be ideal as we require the review of the Kubernetes Output pertaining to the Engine Pod along with Resource Profile & other related artefacts.    Regards, Smarak ... View more

Hello @pankshiv1809    Since we haven't heard from your side concerning the Post, We are marking the Post as Solved. If you have any further ask, Feel free to update the Post & we shall get back to you accordingly.   Regards, Smarak ... View more

Hello @mingtian    Note that Debug Logging won't ensure the Balancer would perform Region Movement, rather the same would confirm if Balancer is running yet the same isn't moving any Region owing to CostFactor. Example: I ensured 1 RegionServer didn't had any Region by RegionMovement & triggered a Balancer, which showed [1] & trigger a Region Movement (Note "Found A Solution That Moves 1 Region"). After the 1st Balancer is Completed, I triggered a 2nd Balancer, which printed [2], wherein the DEBUG report "Skipping Load Balancing".    I believe your Team would see [2] i.e. Balancer is Skipping any Load Balancing owing to Cost Factor. As such, Your Team can consider the fact that HBase is rejecting Region Movement owing to the fact that any new Region-Movement is "Costlier" than Current Region Placement. Tweaking [3] Cost Parameters including setting " hbase.master.loadbalance.bytable" to "true" should help trigger a Balancer for your Team.   Regards, Smarak   [1]  2023-01-18 06:38:33,290 INFO org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished computing new moving plan. Computation took 95 ms to try 7200 different iterations. Found a solution that moves 1 regions; Going from a computed imbalance of 0.4961380973335763 to a new imbalance of 0.020487264673311183. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (not needed); MoveCostFunction : (multiplier=7.0, imbalance=0.3333333333333333, need balance); ServerLocalityCostFunction : (multiplier=25.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction : (not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=1.0, need balance); WriteRequestCostFunction : (multiplier=5.0, imbalance=1.0, need balance); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  [2]  2023-01-18 06:39:05,365 INFO org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Cluster wide - skipping load balancing because weighted average imbalance=0.013858568086431631 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (not needed); MoveCostFunction : (multiplier=7.0, imbalance=0.0); ServerLocalityCostFunction : (multiplier=25.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction : (not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.6685715976063684, need balance); WriteRequestCostFunction : (multiplier=5.0, imbalance=1.0, need balance); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);   [3] StochasticLoadBalancer (Apache HBase 3.0.0-alpha-4-SNAPSHOT API) ... View more

Hello @Ryan_2002    Thanks for engaging Cloudera Community. First of all, Thank You for the detailed description of the Problem. I believe your ask is Valid, yet reviewing the same over a Community Post isn't a suitable approach. Feasible for you to engage Cloudera Support to allow our Team to work with you, with the suitability of Screen-Sharing Session as well as Logs exchange, both of which aren't feasible in Community. That would greatly expedite the review of your ask.    Regards, Smarak ... View more

Hello @mingtian    Hope you are doing well. We wish to follow-up with you & check if the DEBUG Logging assisted in confirming the reasoning for Balancer Algorithm deciding against Region-Movement. If Yes, Kindly let us know if your Q in the Post has been answered or any further Q remains.    Regards, Smarak ... View more

Hello @pankshiv1809    Hope you are doing well. We wish to follow-up on the Post & confirm whether your Team was requesting information into  Dynamic Allocation to allow Spark adjust resources based on Workload requirement.    Regards, Smarak   [1] Job Scheduling - Spark 3.3.1 Documentation (apache.org) ... View more

Hello @panb    Thanks for using Cloudera Community. As far as I am aware, your Team needs to meet the requirement as stated in [1], which doesn't differentiate in Processor Type & I believe your Team is referring to  Hygon Dhyana Processor. Note that we have shared the Hardware requirement is shared for CDP v7.1.8 as CDH isn't recommended now owing to End-Of-Life.    Regards, Smarak   [1] Hardware Requirements | CDP Private Cloud (cloudera.com) ... View more

Hello @mingtian    Thanks for using Cloudera Community. Based on your Post, We would suggest enabling DEBUG Logging for HMaster (Via HMaster UI To Avoid Any Restart) & trigger the Balancer. Generally, Balancer Algorithm may be deciding against running any Region-Alignment owing to Cost Factor [1]. The HMaster Debug Log would print such Balancer information for your review, upon which the Params discussed in [1] can be tuned to force Balancer, yet the Default Params are generally persisted for most Use-Cases.    Note that Balancer Job isn't to merely fit Equal Regions per RegionServer. Balancer consider various Cost as defined by [1] to proceed with Region-Alignment.    Regards, Smarak   [1] StochasticLoadBalancer (Apache HBase 3.0.0-alpha-4-SNAPSHOT API) ... View more

Hello @sachin_saju    Thanks for using Cloudera Community. You have 2 ask in the Post: 1. How to configure different Storage Policies with Cold & Hot Data, 2. Applying different Compression Algorithm in 1 Column Family.    For Q2, I believe the same isn't feasible i.e. Compression Algorithm can be set at CF level. Review [1] for the Compression Algorithm recommendation around Hot & Cold type data.  For Q1, I assume you are referring to HDFS Storage Policy. If Yes, the same is configured uniformly i.e. I am not sure if we can apply different HDFS Storage Policy for different data within the same CF. In HBase, We generally recommend SSD [2] for WAL, else the HBase Data relies on HDFS Storage Policy used. Alternatively, Use BackUp-Restore [3] for having a "Cold" Version of Data, which can be restored as per requirement.    Regards, Smarak   [1] https://hbase.apache.org/book.html#data.block.encoding.types [2] https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/configuring-hbase/topics/hbase-configure-storage-policy-wal.html [3] https://hbase.apache.org/book.html#br.overview     ... View more

Hello @sachin_saju    Thanks for using Cloudera Community. Your queries concerning the Read Path is discussed between a fellow Community User & myself in [1]. Kindly review the same & let us know if the same answer the queries around Read Path. In Summary, Read Path relies on a Merge of BlockCache & MemStore prior to returning the Output to the End-User, thereby avoiding any Inconsistent Read. Refer [2] for few Diagram around the same to help explain the Read Merge Path.    Concerning Doubt # 3, Our Community User asked a similar Q in [3]. I haven't reviewed this Use-Case internally around Hit/Miss Ratio in the UI to answer the same. Henceforth, I shall let our fellow HBase Engineers to answer [3], which may answer your Q3 as well.    Barring Q3, Let me know if your first 2 queries are addressed by [1] & [2].    Regards, Smarak   [1] https://community.cloudera.com/t5/Support-Questions/Is-it-possible-for-inconsistent-read-in-Hbase-with-Memcache/m-p/359452#M238123 [2] https://nag-9-s.gitbook.io/hbase/hbase-architecture/hbase-read-merge [3] https://community.cloudera.com/t5/Support-Questions/Create-cache-miss-scenario-in-HBase-with-HDP-2-6-5/m-p/359795     ... View more

Hello @Girija    Internally, I wasn't able to replicate the issue being faced by you. I was able to create the ConfigSet using "_default" ConfigSet as baseConfig. I am assuming the issue is specific to your Environment & your team should use the CLI to better diagnose such issue. Your team can use the below solrctl command to create a ConfigSet with Solr KeyTab: solrctl config --create Test_Config _default -p configSetProp.immutable=false Assuming the above Command fails, Running the solrctl command with "--trace" after "solrctl" & before "config" would print the trace logging & assist in troubleshooting the issue faced by your team.    Regards, Smarak  ... View more

Hello @brajeshreddy    Since the Issue isn't replicated with CML release internally & your team have engaged Cloudera Support for further assistance, We shall Close the Post now.    For our fellow Community Users, the Steps to perform the Team's Name Modification is shared above. Assuming the same isn't working for Customers, Ensure you are connected as MLAdmin & Caching is ruled out as well.    Regards, Smarak ... View more

Hello @sekhar1    Since we haven't heard back from your side concerning the Post, We shall mark the Post as Resolved with the Action Plan to review if  Traffic from your Machine isn't allowed to or from the Security Group linked with the VPC wherein the CML Workspace instances are deployed. Check if the Kubernetes Pods associated with the CML Workspace Kubernetes Cluster are Up/Running. If Yes, Such Exception should be reviewed from a Network Standpoint only.    You may reach out to your Customer's AWS/Platform Team to review the Traffic between your Machine & the VPC within which the CML Workspace is deployed.    Assuming your team fixed the issue outside of any Customer's Network concerns, We would appreciate your feedback to ensure our fellow Community Users can benefit from your experience.    Regards, Smarak ... View more

Hello @SagarCapG    Thanks for using Cloudera Community. While I haven't tested the above Use-Case in Phoenix v5.0.0, I tested in Phoenix v5.1.0 (Used in CDP v7.1.7) & the same shows the Primary Key correctly [1]. As such, I believe the above ask is fixed in CDP v7.1.8 atleast.    I reviewed Upstream JIRA & it appears the same is actually fixed in Phoenix v5.1.x as per comments in https://issues.apache.org/jira/browse/PHOENIX-6651. As such, Kindly plan for CDP v7.1.8 Upgrade for obtaining the concerned functionality of identifying PK via " !primarykeys".    Regards, Smarak   [1]  0: jdbc:phoenix:> CREATE TABLE Employee (EmpId Integer Not Null Primary Key, Ename Varchar(50)); 0: jdbc:phoenix:> !primarykeys Employee; +-----------+-------------+------------+-------------+---------+---------+-------------+-----------+-----------+-------------+---------+---------------+ | TABLE_CAT | TABLE_SCHEM | TABLE_NAME | COLUMN_NAME | KEY_SEQ | PK_NAME | ASC_OR_DESC | DATA_TYPE | TYPE_NAME | COLUMN_SIZE | TYPE_ID | VIEW_CONSTANT | +-----------+-------------+------------+-------------+---------+---------+-------------+-----------+-----------+-------------+---------+---------------+ | | | EMPLOYEE | EMPID | 1 | | A | 4 | INTEGER | null | 4 | | +-----------+-------------+------------+-------------+---------+---------+-------------+-----------+-----------+-------------+---------+---------------+   ... View more

Hello @anks1    Thanks for using Cloudera Community. Kindly confirm the following details:   CML Deployment (CDP Public Cloud Or Private Cloud) CML Version Used (Via CML Workspace Details In CML Page) Any timeframe of how many days before an App transition to Failed State from Running State   Regards, Smarak ... View more

Hello @sekhar1    We wish to follow-up on the Post & confirm if your Team have resolved the issue. In Summary,  such Exception are received if Traffic from your Machine isn't allowed to or from the Security Group linked with the VPC wherein the CML Workspace instances are deployed. Check if the Kubernetes Pods associated with the CML Workspace Kubernetes Cluster are Up/Running. If Yes, Such Exception should be reviewed from a Network Standpoint only.    You may reach out to your Customer's AWS/Platform Team to review the Traffic between your Machine & the VPC within which the CML Workspace is deployed.    Regards, Smarak ... View more

Hi @pacman A Merge happens for each Read Operation i.e. BlockCache & MemStore. As such, Incorrect Values aren't observed. Having said that, If you observe any such scenario of Read/Write Inconsistency, Kindly share a Use-Case & any replication attempt to allow us to review accordingly.   Regards, Smarak ... View more

Hello @pacman    Thanks for using Cloudera Community. HBase handles such scenario by Read Merge. A Read "Merges" Key Values from the Block Cache, MemStore, and HFiles in the following steps:   First, the Scanner looks for the Row Cells in theBlock Cache (Read Cache).  Next, the Scanner looks in the MemStore (Write Cache). If the Scanner does not find all of the Row Cells in the MemStore and Block Cache, then Hfiles are referred.    Hope the above answers your query. If Yes, Kindly mark the Post as Resolved.    Regards, Smarak ... View more