Hello @kras    I was able to reproduce the concerned behaviour as well. Checking internally on the Steps to safely remove the Secondary Regions. Shall Update the Post once a Solution is found.   - Smarak ... View more

Hello @Manoj690    The best way would be to check the RegionServer Logs wherein the Region are stuck in OPENING State. The fact that Count of Region-In-Transition is reducing with time & Increasing again upon HBase Service Restart indicates the Regions are likely waiting on WAL Split & Edit Replay. With 3 RegionServer, 7K RIT indicates you must be having more than 7K Regions in Total. As such, the RegionServer are overloaded & WAL Split & Edit Replay would be Slow.    There are RegionOpening Thread, which can be Increased yet the Ideal Solution would be to Reduce the RegionCount or Increase RegionServer Count to ~300-400 Region/RegionServer.    - Smarak ... View more

Hi @kumarkeshav    Your Observation is Correct. The Config isn't exposed via Ambari v2.7.x yet adding the same via Custom HBase-Site.XML is prohibited with the "This property is already defined in this configuration file" message. The same isn't an issue with Ambari v2.6.x, wherein the concerned Write Limit is exposed.    I am not familiar with the Exact Location in Ambari DB, wherein the concerned Parameter is exposed to be manually over-written. The Simplest Solution is the one you are possibly using i.e. Edit the XML for the RegionServer directly & restart them manually to ensure they are picking up the concerned Updated Configuration. However, this isn't practical in a Large Env unless an Automation Tool is used.    If I come across an Easier Fix, I shall update you in the Post.    - Smarak ... View more

Hello @MG-1    The Symptoms appears to match HBASE-20616.   In your Case, You should follow the below Steps: (I) Confirm the only RUNNABLE Procedures are TruncateTableProcedure (II) Stop the HMaster Services (Active & Standby).  (III) Sideline the Contents of MasterProcWALs Directory (/hbase/MasterProcWALs). (IV) Sideline the Table Region Directories from "/hbase/.tmp/" Directory.  (V) Start the HMaster Services.    The above Process ensures the Master doesn't attempt the Truncate Table Procedures by sidelining the MasterProcWALs Directory Contents & Sideline the Table Region Directories from "/hbase/.tmp/" Directory, else the Table won't be allowed to be Created again.    - Smarak ... View more

Very well-written article. ... View more

I am assuming the above change wouldn't survive future Ambari upgrade. ... View more

Thanks Brandon. I used the above approach to retrieve the history of alerts for a specific type. Adding a small trick for anyone using the above approach. The table "Alert_History" has the Alert Timestamp column in BigInt. For readability, use the PSQL Cast functionality. In my example, I am using the SQL to identify when Ambari triggers the DataNode WebUI alerts across all DataNodes: Select TO_CHAR(TO_TIMESTAMP(Alert_Timestamp / 1000), 'MM/DD/YYYY HH24:MI:SS') From Alert_History Where Alert_Label Like '%DataNode%UI%' And Alert_State ='CRITICAL' Order By 1 ASC; ... View more

Command To Check SafeMode On/Off: hdfs dfsadmin -safemode get Command To Get Out Of Safe Mode Shared Above. ... View more

The 04-Series article are very well written. Thanks Arpit for documenting them. ... View more