Hello @kvinod    As Cluster Replication wasn't being used based on the fact that "list_peers" isn't showing any Peer, It's likely the CleanerChore Thread wasn't performing its duties. Note that WALs are moved to oldWALs once the Last SequenceIDs of the WALs have been persisted to Disk via MemStore Flush. In other words, oldWALs being present doesn't necessarily means that the WALs are being persisted for replication. Now, the Cleanup of oldWALs is CleanerChore Thread responsibility. As we covered above, the HBase Service Restart covered the HMaster Restart, which would ensure the CleanerChore Thread is spawned afresh.    Let me know if the above answers your queries.   - Smarak ... View more

Hello @tuk    If the Post by Pabitra assisted you, Kindly mark the Post as Solution. If you utilised any other approach, Kindly share the details in the post as well.   Thanks, Smarak ... View more

Hello @TGH    Sharing the Steps for building the HBCK2 Jar using Git reference & additionally, refer the Post via [1] for the details on building HBCK2 Tool as well.     - Smarak   [1] https://community.cloudera.com/t5/Support-Questions/How-to-get-hbck2-tool-for-CDH-6-3-2/m-p/295867/highlight/true#M218004 ... View more

Hello @TGH    Thanks for using Cloudera Community. You had Region-In-Transition (RIT) & the HDFS Directory has been removed for the Regions along with the ZNode being removed, yet HBase reports RIT. You wish to fix the RIT issue by removing the Meta Table entries as RIT avoids Balancer run. In HBase v2 (CDH v6.3.x), the MasterProcWALs is critical for any Procedure, which are stuck or blocked. You mentioned a lot of procedures (Disable|Delete) being observed.    The graceful manner for your Team to manage the requirement is to use the HBCK2 Tool. You can build the HBCK2 using the Link [1]. Next, You can use the HBCK2 Tool to bypass the Procedure (PIDs) associated with the Table, for which the Region Directories have been removed. Once any PID is bypassed, the HMaster UI Page (Locks & Procedures ) Section would show the PID as "Bypass". After ensuring the required PIDs are bypassed, Restart the HMaster Service & use the HBCK2 Tool to remove the Region entries in Meta, for which the HDFS Region Directories are removed. Use "bypass" & "extraRegionsInMeta" HBCK2 Command as documented in Link [1].   Alternatively, You can Stop HMaster > Remove MasterProcWALs (After confirming no RUNNABLE Procedures excluding the PID associated with the Table for which Region Directory have been removed) > Start HMaster. However, this isn't an ideal approach & you can encounter "Master Is Initialising" issue, for which HBCK2 Tool is required. The "Master Is Initialising" context is captured in Link [1] as well.   - Smarak   [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2 ... View more

Hello @Manoj690    Thanks for using Cloudera Community. Your concern is Phoenix Table created on top of a Restored Table isn't showing the Non-PrimaryKey Columns correctly.   Can you share the Steps used by your Team to backup the Table & subsequent restore. Additionally, Whether the Backup & Restore are being performed within the same Cluster along with the Distribution being used (For Versioning Check).   - Smarak  ... View more

Hello @ibr    If you are referring to the Supported Database for CDP Private Cloud Base (Which is the SDX for Private Cloud Experience), the list of Supported Databases for CDP Private Cloud Base, wherein the Databases for Metadata, Authorisation is maintained is shared in the Link [1].    [1] https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdpdc-database-requirements.html ... View more

Hello @Manoj690    RegionServer is a Service & your team can add the RegionServer Service interactively using via Ambari (HDP) or Cloudera Manager (CDH or CDP).   - Smarak ... View more

Hello @lenu    If you have Replication enabled, WALs are likely to be persisted until the WALs are replicated. If you aren't using HBase Replication, Ensure there are no Peers (via "list_peers") & " hbase.replication" Property is false. If the oldWALs aren't removed, Enable TRACE Logging for the HBase Master Service, which would print the CleanerChore Thread removing or skipping any entries.    - Smarak   ... View more

Hello @phongnamam    To your Queries, (I) Split tries to take an approximate centre point of the RowKey for the Region being split. In this manner, Yes, Region Split is a Horizontal Split. By default, each Table would have 1 Regions. You mentioned, you have 3 Tables & 5 Regions. The same is feasible as each Table have 1 Region (Total 3 Regions) & the Hbase:Meta/Hbase:NameSpace Table have 1 Region each, making total 5 Regions. If a Small Table have 5 Regions, It's likely using Split Point.    (II) When a RegionServer dies, the Regions hosted on the RegionServer are distributed across Live RegionServers & when the RegionServer comes back online, the Master assign Regions to the RegionServer based on Balancer. The Master Logs would show if Balancer is moving Regions as part of Balancing.    (III) Ideally, Masters shouldn't flip if 1 RegionServer stops. Best approach is to check the Logs of both Masters to see for any issues.   - Smarak ... View more

Hello @TR7_BRYLE    It's an old Post, yet trying to answer the Query. If the eventual goal of splitting 1 Snapshot into 3 Snapshots is to Merge them, We would like to know your Use-Case & what your Team is trying to achieve. Splitting Snapshot isn't feasible, yet there may be other choices to achieve the Use-Case your Team is achieve, if your Team can share the same.   - Smarak ... View more

Hello @shubs    If you have fixed the issue, Kindly update the Post with the Solution.   The Exception is coming from  DFSInputStream with EOFException. Worth checking if the RegionServer having issues has any Zero Length WAL File under the WAL Directory. Or, Enable TRACE Logging on the concerned RegionServer to capture additional details on the concerned Exception. Or, Checking which WAL is being replicated & any FS issues with the blocks associated with the WAL File.   - Smarak ... View more

Hello @SimL    YARN Timeline Server v2 Reader uses HBase for Storage. The Log & HBCK Report shows the Table "prod.timelineservice.entity" has Regions in OPENING State, which would naturally cause any Client accessing the Table report RetriesExhaustedWithDetailsException.   A Region is typically stuck in OPENING State, if there is any issues with WALEditReplay or any other reason, which would be clear from the RegionServer Logs where the Region is being opened. Unless the RegionServer Logs are available, any Comment is unlikely to be an accurate assessment of the issue your team is facing. Would recommend checking the RegionServer Logs wherein the Regions are in OPENING State & based on the cause, plan accordingly.    - Smarak ... View more

Hello @qu1etstorm    The Logs  FileNotFoundException & YouAreDeadException are typically observed after the Cause, so worth checking the Logs before the concerned 2 Exception. TheYouAreDeadException is thrown by the Master when a RegionServer reports and is already being processed as Dead. This can happen when aRegionServer loses its session but didn't figure it yet. So, Best to check with respect to the RegionServer ZNode Session tracking in the Logs, while comparing the time at when the Master detected the RegionServer Ephemeral ZNode as being removed. Typically, JVMPause are the most likely cause.    - Smarak ... View more

Hello @amit_    The Post is couple of months old, yet wish to know if you have fixed the issue. The Lease seems to have expired after ~30 Minutes as per your Configuration. However, the Log shows a processingTime of ~30/~60 Seconds for ~5MB of Scan Size, so the Scanner Lease expiring after ~30 Minutes looks strange & appears to be caused by either the Client going away somehow or the processingTime being excessively slow.    If you have fixed the issue, Will appreciate your feedback on the Post.   - Smarak  ... View more

Thank You @Seaport for sharing the Solution for wider audience. In short, you followed the Link [1] to resolve the Protocol Header Issue as utilised by the User in Link [2].    - Smarak   [1] https://github.com/python-happybase/happybase/issues/161 [2] https://community.cloudera.com/t5/Support-Questions/Sharing-how-to-solve-HUE-and-HBase-connect-problem-on-CDH-6/td-p/82030   ... View more

Hello @jiaozz    It's an Old Post, yet submitting a response to anyone encountering such issue. Yet, Kindly test the 2 approaches below: (I) At Hive Session Level, Execute " ADD   FILE  <HBase Configuration File> " (II) Add the HBase Config to "HIVE_AUX_JARS_PATH" Path at Hive Service Level.    In short, the issue appears upon HBase Config being not correctly set, even though the 1st Create Table Hive SQL runs successfully yet fails upon any Table created referencing the Hive Table created in the 1st SQL.    - Smarak ... View more

Hello @ASIF123    Atlas Janus is an HBase Table, created with Column Family specification only. The DDL of the concerned Atlas Table is available in the HMaster UI & Master Logs, when Atlas Service is initialised for 1st time. However, Column Families are added eventually & each Row of the concerned Table has Different Column Qualifier. In short, there is no Static Definition of Atlas Janus Table.    Any Phoenix Table or View mapping requires explicit specification of the Column Family & Column Qualifier. Henceforth, the Phoenix Table or View DDL will be specific to Customer's Env. Your team can review the Atlas Janus Table Output via Scan & confirm the Column Qualifier against each Column Family. Accordingly, the Phoenix Table or View can be created.    - Smarak ... View more

Hello @rameshlk    Maybe this Post by Josh Elser would help, wherein he discussed using PQS behind Load Balancer to achieve High Availability: https://community.cloudera.com/t5/Community-Articles/Deploying-the-Phoenix-Query-Server-in-production/tac-p/245110   - Smarak ... View more

Hello @kras    I was able to reproduce the concerned behaviour as well. Checking internally on the Steps to safely remove the Secondary Regions. Shall Update the Post once a Solution is found.   - Smarak ... View more

Hello @Manoj690    The best way would be to check the RegionServer Logs wherein the Region are stuck in OPENING State. The fact that Count of Region-In-Transition is reducing with time & Increasing again upon HBase Service Restart indicates the Regions are likely waiting on WAL Split & Edit Replay. With 3 RegionServer, 7K RIT indicates you must be having more than 7K Regions in Total. As such, the RegionServer are overloaded & WAL Split & Edit Replay would be Slow.    There are RegionOpening Thread, which can be Increased yet the Ideal Solution would be to Reduce the RegionCount or Increase RegionServer Count to ~300-400 Region/RegionServer.    - Smarak ... View more

Hi @kumarkeshav    Your Observation is Correct. The Config isn't exposed via Ambari v2.7.x yet adding the same via Custom HBase-Site.XML is prohibited with the " This property is already defined in this configuration file" message. The same isn't an issue with Ambari v2.6.x, wherein the concerned Write Limit is exposed.    I am not familiar with the Exact Location in Ambari DB, wherein the concerned Parameter is exposed to be manually over-written. The Simplest Solution is the one you are possibly using i.e. Edit the XML for the RegionServer directly & restart them manually to ensure they are picking up the concerned Updated Configuration. However, this isn't practical in a Large Env unless an Automation Tool is used.    If I come across an Easier Fix, I shall update you in the Post.    - Smarak ... View more

Hello @MG-1    The Symptoms appears to match HBASE-20616.   In your Case, You should follow the below Steps: (I) Confirm the only RUNNABLE Procedures are TruncateTableProcedure (II) Stop the HMaster Services (Active & Standby).  (III) Sideline the Contents of MasterProcWALs Directory (/hbase/MasterProcWALs). (IV) Sideline the Table Region Directories from " /hbase/.tmp/" Directory.  (V) Start the HMaster Services.    The above Process ensures the Master doesn't attempt the Truncate Table Procedures by sidelining the MasterProcWALs Directory Contents & Sideline the Table Region Directories from "/hbase/.tmp/" Directory, else the Table won't be allowed to be Created again.    - Smarak ... View more

Review Link "https://doc.lucidworks.com/lucidworks-hdpsearch/2.6/Guide-Install-Ambari.html" Section "Advanced solr-metrics". ... View more