Hello @xgxshtc    We observed you have posted the concerned ask in a New Post [1] as the concerned Post is ~4Years Old. While the Current Post is Unresolved, We shall wait on your Team's review on [1] before confirming the Solution on the Current Post as well.   Regards, Smarak   [1] https://community.cloudera.com/t5/Support-Questions/Hbase-regionserver-shutdown-after-few-hours/m-p/330070/highlight/false#M230589   ... View more

Hello @xgxshtc    Thanks for using Cloudera Community. Based on the Post, RegionServer is reporting EOFException | "Bad DataNode" while replaying WALs "/hbase/WALs". It appears the HDFS Blocks are having issues. To fix the Issue, Your Team can Sideline the Contents of the "/hbase/WALs" (Specific "/hbase/WALs/jfhbase03,60020,1636456250380") & restart the concerned RegionServer. If all RegionServers are Impacted, Sideline each of the RegionServer Directories from "/hbase/WALs". Note that the WALs hold the Edits not yet persisted to Disk & Sidelining the WAL Directories (1 WAL per RegionServer) may incur Data Loss.    Additionally, Review the HDFS FSCK of HDFS files with WALs "/hbase/WALs" & fix any Corrupt/Missing Blocks. After ensuring the HDFS FSCK of WALs "/hbase/WALs" is Healthy, Your Team can restart the HBase RegionServers.    Regards, Smarak ... View more

Hello @Faizan_Ali    Thanks for using Cloudera Community. Based on the Post, ZooKeeper Service is failing to start on CDH v5.16. The Screenshot covers the "stdout" file. Kindly share the "stderr" file or the "Role Log" file for reviewing the issue with ZooKeeper Service.    If your Team have Solved the Post, Kindly share the Issue & Steps performed to Mitigate the Issue. Your assistance would help our fellow Community Users.    Regards, Smarak ... View more

Hello @yacine_    Thanks for sharing the Solution along with the Root Cause as well. We shall mark the Post as Solved for now to ensure fellow Community Users can use the Solution as well.    Regards, Smarak ... View more

Hello @vishal6196    Thanks for using Cloudera Community. Based on the Post, the Phoenix Delete Command is reporting more records being deleted than the Select Count(*) returns. Thank You for sharing the HDP & Phoenix Version as well.    Few Queries for your Team: Whether the Observation is made for all Tables or Selective Table(s), Explain Output of the Select & Delete SQL, Any Index of the Phoenix Table being deleted, If Performing MajorCompaction on the Table before Deleting (Just for Sanity Check) shows any Difference with respect to the Rows Deleted via Delete SQL.    Regards, Smarak ... View more

Hello @SteffenMangold    Kindly let us know if the Post was Helpful for your Team. If Yes, Please mark the Post as Solved. If No, Do share the Outcome of the Command shared in our Post. Accordingly, We shall review & get back to you.   Regards, Smarak ... View more

Hello @Ma_FF    Kindly let us know if the Post was Helpful to Identify the RegionServer Thread spiking the CPU Usage of the RegionServer JVM. If Yes, Please mark the Post as Solved.    Regards, Smarak ... View more

Hello @paresh    Thanks for using Cloudera Community. Based on the Post, HBase Service is being impacted owing to the Trace shared by you in the Post with the Solutions attempted. While Complete Logs helps, It appears the Region "1595e783b53d99cd5eef43b6debb2682" Replay via "recovered.wals" is being interrupted owing to EOFException.   When a Region is being Opened, any Contents in the "recovered.wals" are Replayed by reading them & pushing to MemStore. Once the Edits are persisted from MemStore to an Hfile, the "recovered.wals" are Removed.    With the Possibility of DataLoss, You may attempt to Stop HBase > Sideline the RecoveredEdits "hdfs://ha-cluster:8020/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals" & MasterProcWALs (To avoid any Replay of associated PID) > Start HBase > Verify the Status. Note the "DataLoss" is likely from removing the RecoveredEdits File not persisted yet.    Additionally, You may use WALPlayer [1] for replaying the Contents of RecoveredEdits as well.    Regards, Smarak   [1] https://hbase.apache.org/book.html#walplayer   ... View more