Hello @vishal6196    Thanks for using Cloudera Community. Based on the Post, the Phoenix Delete Command is reporting more records being deleted than the Select Count(*) returns. Thank You for sharing the HDP & Phoenix Version as well.    Few Queries for your Team: Whether the Observation is made for all Tables or Selective Table(s), Explain Output of the Select & Delete SQL, Any Index of the Phoenix Table being deleted, If Performing MajorCompaction on the Table before Deleting (Just for Sanity Check) shows any Difference with respect to the Rows Deleted via Delete SQL.    Regards, Smarak ... View more

Hello @SteffenMangold    Kindly let us know if the Post was Helpful for your Team. If Yes, Please mark the Post as Solved. If No, Do share the Outcome of the Command shared in our Post. Accordingly, We shall review & get back to you.   Regards, Smarak ... View more

Hello @Ma_FF    Kindly let us know if the Post was Helpful to Identify the RegionServer Thread spiking the CPU Usage of the RegionServer JVM. If Yes, Please mark the Post as Solved.    Regards, Smarak ... View more

Hello @paresh    Thanks for using Cloudera Community. Based on the Post, HBase Service is being impacted owing to the Trace shared by you in the Post with the Solutions attempted. While Complete Logs helps, It appears the Region " 1595e783b53d99cd5eef43b6debb2682" Replay via "recovered.wals" is being interrupted owing to EOFException.   When a Region is being Opened, any Contents in the "recovered.wals" are Replayed by reading them & pushing to MemStore. Once the Edits are persisted from MemStore to an Hfile, the "recovered.wals" are Removed.    With the Possibility of DataLoss, You may attempt to Stop HBase > Sideline the RecoveredEdits "hdfs://ha-cluster:8020/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals" & MasterProcWALs (To avoid any Replay of associated PID) > Start HBase > Verify the Status. Note the "DataLoss" is likely from removing the RecoveredEdits File not persisted yet.    Additionally, You may use WALPlayer [1] for replaying the Contents of RecoveredEdits as well.    Regards, Smarak   [1] https://hbase.apache.org/book.html#walplayer   ... View more

Hello @Malvin    We are marking the Post as Solved as Region Server Grouping is the Only Way to Segregate the Region Assignment at Region Server Level. If your Team encounter any issues, Feel free to Update the Post & we shall get back to your Team accordingly.   Thank You again for using Cloudera Community.   Regards, Smarak ... View more

Hello @rahuledavalath    We are marking the Post as Solved. If your Team have any further concerns, Please update the Post, if your Team face any issues with either of the Recommendation shared [Migrate the System Tables Or Create a Phoenix Table afresh on Top of HBase Table].   Thank You for using Cloudera Community.    Regards, Smarak ... View more

Hello @Ma_FF    Thanks for using Cloudera Community. Based on the Post, 1 Region Server is using High CPU. As requested by @PrathapKumar, Review the same. Additionally, Your Team can perform the below:   (I) When the Region Server JVM reports High CPU, Open "top" Command for the Region Server PID, (II) Use "Shift H" to open the Thread View of the PID. This would show the Threads within the Region Server JVM with CPU Usage, (III) Monitor the Thread View & Identify the Thread hitting the Max CPU Usage, (IV) Take Thread Dump | JStack of Region Server PID & Compare the Thread with the "top" Thread View consuming the Highest CPU.    The above Process would allow you to identify the Thread contributing towards the CPU Usage. Compare the same with other Region Server & your Team can make a Conclusive Call to identify the reasoning for CPU Utilization. Howsoever Logs are reviewed, Narrowing the Focus of JVM review would assist in identifying the Cause. Review shared Link for additional reference.   Kindly review & share your Observation in the Post.    Regards, Smarak   [1] https://www.infoworld.com/article/3336222/java-challengers-6-thread-behavior-in-the-jvm.html [2] https://blogs.manageengine.com/application-performance-2/appmanager/2011/02/09/identify-java-code-consuming-high-cpu-in-linux-linking-jvm-thread-and-linux-pid.html [3] https://blog.jamesdbloom.com/JVMInternals.html   ... View more

Hello @amit_hadoop    Thanks for using Cloudera Community. Based on the Post, You installed HDP Sandbox 3.0 & observed Timeline Service v2 Reader reporting [1].    If you observe the Port mentioned, It says "17020" & HBase RegionServer runs with "16020" Port by default. YARN Timeline Service uses an Embedded HBase Service, which isn't equivalent to HBase Service as we may see in the Ambari UI. Kindly let us know the Outcome of the below Steps:   Whether Restarting YARN Timeline Service (Which restarts the Embedded HBase Service) helps, Within the YARN Logs, We should have an Embedded HBase Logs for HMaster & RegionServer. Reviewing the Logs may offer additional clue into the reasoning for Embedded HBase concerns, Review [2] for similar concerns wherein the ZNode was Cleared & Service Restarted, Review [3] for Documentation on Deleting the ZNode & Data. Restarting the YARN Timeline Service should initialise the HBase Setup afresh.    Regards, Smarak   [1] Error Trace: 2021-09-29 05:47:58,587 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=5385 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server sandbox-hdp.hortonworks.com,17020,1632894462010 is not running yet details=row 'prod.timelineservice.entity' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=sandbox-hdp.hortonworks.com,17020,1628801419880, seqNum=-1 [2] https://community.cloudera.com/t5/Support-Questions/Yarn-Timeline-Service-V2-not-starting/td-p/281368 [3] https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/data-operating-system/content/remove_ats_hbase_before_switching_between_clusters.html     ... View more

Hello @SteffenMangold    Thanks for using Cloudera Community. Based on the Post, Your Team wish to assign a Region (Which isn't assigned to any RegionServer) without any HBase Restart.   We wish to confirm whether your Team tried using the following approaches: HBCK2 Assign as documented via [1]. The Command requires the Region ID as argument & returns a PID, which can be viewed via HMaster UI "Locks & Procedures" or "list_procedures" via HBase Shell for Success/Failure. Within HBase Shell, There are "assigns" Or "move" Command allowing Destination Region Server to be specified as well.  Your Team may have tried the above Command yet the Outcome of the above Command would confirm if there is any Underlying concerns with the Region, which requires additional intervention like HBase Service Restart (Or, HMaster Restart which should spawn an Assignment Manager Thread afresh).    Regards, Smarak   [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#running-hbck2 ... View more

Hello @Sayed016    Thanks for using Cloudera Community. Based on the Post, Your Team is experiencing Exception [1] for RangerAudits Shard1 Replica1, followed by Successful Connect with ZooKeeper to eventually failing with [2]. Once the same happened, What is being observed by your Team i.e. Whether the Collection RangerAudits Shard1 Replica1 enters "Down" State as opposed to "Active" State.    Since the Logs shows RangerAudits Shard1 has no Replica, It's feasible the Issue arises from Consistency concerns between Solr & ZooKeeper. There are few things we wish to verify with your assistance: When Solr reports [1], Whether the ZooKeeper Quorum are Healthy Or, any issues with the ZooKeeper Server wherein the Solr ZooKeeperClient is connected, What happens after [2] for RangerAudits Shard1 Replica1. This is requested above as well, Whether Restarting Solr Service on Host wherein "ranger_audits_shard1_replica_n1" Core is hosted helps in mitigating the "ClusterState Says We Are Leader, But Locally We Don't Think So".  The HDP, CDH, CDP Version being discussed in the Post.  Regards, Smarak   [1] 2021-10-17 04:05:57.006 ERROR (qtp1916575798-2477) [c:ranger_audits s:shard1 r:core_node2 x:ranger_audits_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are disabled.   [2] 2021-10-17 04:14:37.504 ERROR (qtp1916575798-2325) [c:ranger_audits s:shard1 r:core_node2 x:ranger_audits_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: ClusterState says we are the leader (https://192.168.0.17:8985/solr/ranger_audits_shard1_replica_n1), but locally we don't think so. Request came from null ... View more

Hello @JB0000000000001    As we haven't heard from your side, We assume the Queries posted by you has been addressed & marking the Post as Solved. When you have the time, Feel free to share your  Observation with respect to studying or implementing HBase on Cloud Storage.    Thanks again for sharing your thoughts on Cloudera Community.   - Smarak ... View more

Follow-Update to tag @KR_IQ @sppandita85BLR as the Original Post is Old.  ... View more

Hello @michalm_    Thanks for using Cloudera Community. While I haven't performed such Task, I wish to check if you have reviewed the 3rd Party Script via [1], which uses a Use-Defined Time to find Impala Queries & optionally, Kill them as well. Let us know if it helps.    - Smarak   [1] https://github.com/onefoursix/kill-long-running-impala-queries   ... View more

Hello @mahfooz-iiitian    Please note that CDH Releases are End-Of-Support & CDP Releases aren't supporting Spark-HBase-Connector in favor of HBase-Spark-Connector as documented below. In short, Hortonworks Spark-HBase Connector isn't Supported yet in favor of HBase Connector for Spark.    -  Smarak   [1] https://issues.apache.org/jira/browse/HBASE-25326 [2] https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_HBase_Connector.md [3] https://docs.cloudera.com/cdp-private-cloud/latest/data-migration/topics/cdp-data-migration-hbase-prepare-data-migration.html   ... View more

Hello @Chandresh    Hope you are doing well. We wish to confirm if you have identified the Cause of the issue. If Yes, Kindly share the same to benefit our fellow Community Users as well. If no further assistance required, Please mark the Post as Solved.    - Smarak ... View more

Hello @ryu    As mentioned by @arunek95, we assume Phoenix is enabled for the Cluster. If not, Kindly enable Phoenix & try the Command again.    The Logging indicates HDP v2.6.1.0 with Phoenix v4.7. The Directory " /usr/lib/phoenix/" has the Phoenix Client & you mentioned the same Directory has Phoenix Server Jar as well. Kindly verify if the Permission on the JAR is Correct & confirm via "jar -tvf" on the Phoenix Server Jar that the Class "MetaDataEndpointImpl" is included in the same.    The Error indicates the Phoenix creating the SYSTEM Tables (Upon 1st Connection to Phoenix) is encountering the Error. In our Internal Setup, We see the Phoenix-Server Jar is present in HBase Lib Path as well, pointing to the Phoenix-Server Jar in Phoenix Lib Path as SymLink: /usr/hdp/<Version>/hbase/lib/phoenix-server.jar -> /usr/hdp/<Version>/phoenix/phoenix-server.jar   Kindly ensure the Phoenix Server JAR is present in HBase Lib Directory as well. Additionally, Review the Master Logs to check for the Error Message at HBase Level as well.    - Smarak ... View more

Hello @gurucgi    This is an Old Post yet we wish to check if the issue has been addressed by you. If Yes, Please share the Steps to ensure fellow Community Users can benefit from your experience.    Based on the Post, SystemCatalogRegionObserver is being reported for Class not being loaded. The concerned Error is being received for Phoenix v5.1.1. In Cloudera, We are shipping with Phoenix v5.1.0 at the time of writing. I am not sure if you are being impacted via [1]. In short, the JAR seems to be requiring the Phoenix-Server Jar to be placed rightfully.    - Smarak   [1] https://issues.apache.org/jira/browse/PHOENIX-6330   ... View more

Hello @Satya_Singh    Do let us know if your issue has been resolved. If Yes, Please share the Mitigation Steps followed by you to ensure other Community Users can benefit from your experience & mark the Post as Resolved as well.    - Smarak ... View more

Hello @JB0000000000001    We wish to follow-up with you on the Post & confirm if you have any additional Observation to be shared with respect to studying or implementing HBase on Cloud Storage. Or, our response to your Post was helpful in getting the required Cloud Storage possible latencies.    - Smarak ... View more