Member since
01-16-2018
540
Posts
33
Kudos Received
82
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
103 | 01-18-2023 12:10 AM | |
69 | 01-16-2023 01:54 AM | |
162 | 01-13-2023 01:59 AM | |
156 | 01-13-2023 01:35 AM | |
93 | 01-02-2023 10:03 PM |
12-08-2020
09:12 AM
1 Kudo
Hello @kvinod As Cluster Replication wasn't being used based on the fact that "list_peers" isn't showing any Peer, It's likely the CleanerChore Thread wasn't performing its duties. Note that WALs are moved to oldWALs once the Last SequenceIDs of the WALs have been persisted to Disk via MemStore Flush. In other words, oldWALs being present doesn't necessarily means that the WALs are being persisted for replication. Now, the Cleanup of oldWALs is CleanerChore Thread responsibility. As we covered above, the HBase Service Restart covered the HMaster Restart, which would ensure the CleanerChore Thread is spawned afresh. Let me know if the above answers your queries. - Smarak
... View more
12-08-2020
06:26 AM
Hello @tuk If the Post by Pabitra assisted you, Kindly mark the Post as Solution. If you utilised any other approach, Kindly share the details in the post as well. Thanks, Smarak
... View more
12-08-2020
06:20 AM
Hello @TGH Sharing the Steps for building the HBCK2 Jar using Git reference & additionally, refer the Post via [1] for the details on building HBCK2 Tool as well. - Smarak [1] https://community.cloudera.com/t5/Support-Questions/How-to-get-hbck2-tool-for-CDH-6-3-2/m-p/295867/highlight/true#M218004
... View more
12-08-2020
06:15 AM
Hello @ma_lie1 This is an Old Post, yet sharing the details to close the Post & for future reference. You can build the HBCK2 Tool from the HBCK2 Git Page. Sharing the Steps below (Expect git & maven to be installed). The Command Usage is documented via Link [1]: - Smarak [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
... View more
12-08-2020
06:05 AM
Hello @TGH Thanks for using Cloudera Community. You had Region-In-Transition (RIT) & the HDFS Directory has been removed for the Regions along with the ZNode being removed, yet HBase reports RIT. You wish to fix the RIT issue by removing the Meta Table entries as RIT avoids Balancer run. In HBase v2 (CDH v6.3.x), the MasterProcWALs is critical for any Procedure, which are stuck or blocked. You mentioned a lot of procedures (Disable|Delete) being observed. The graceful manner for your Team to manage the requirement is to use the HBCK2 Tool. You can build the HBCK2 using the Link [1]. Next, You can use the HBCK2 Tool to bypass the Procedure (PIDs) associated with the Table, for which the Region Directories have been removed. Once any PID is bypassed, the HMaster UI Page (Locks & Procedures ) Section would show the PID as "Bypass". After ensuring the required PIDs are bypassed, Restart the HMaster Service & use the HBCK2 Tool to remove the Region entries in Meta, for which the HDFS Region Directories are removed. Use "bypass" & "extraRegionsInMeta" HBCK2 Command as documented in Link [1]. Alternatively, You can Stop HMaster > Remove MasterProcWALs (After confirming no RUNNABLE Procedures excluding the PID associated with the Table for which Region Directory have been removed) > Start HMaster. However, this isn't an ideal approach & you can encounter "Master Is Initialising" issue, for which HBCK2 Tool is required. The "Master Is Initialising" context is captured in Link [1] as well. - Smarak [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
... View more
12-08-2020
05:48 AM
Hello @kvinod Thanks for the Update. The Replication ZNode being created is expected after restart. The Checkbox concerning HBase Replication being left unchecked indicates Replication being disabled yet I have observed couple of cases wherein a CM Config wasn't passed to Service Level, causing certain unexpected behaviour. The explicit addition of the Parameter was to ensure the Service (HBase in this case) is aware of the Configuration. Or, Master Restart (Performed via HBase Restart) may have resolved the issue, by spawning a new CleanerChoreThread. As such, the issue is likely with the HBase Service being unaware of Replication being disabled or HMaster CleanerChore Thread. By explicitly adding the HBase Replication as False & restarting the HBase Service, We covered the 2 possibilities. - Smarak Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
... View more
12-08-2020
03:29 AM
Hello @Manoj690 Thanks for using Cloudera Community. Your concern is Phoenix Table created on top of a Restored Table isn't showing the Non-PrimaryKey Columns correctly. Can you share the Steps used by your Team to backup the Table & subsequent restore. Additionally, Whether the Backup & Restore are being performed within the same Cluster along with the Distribution being used (For Versioning Check). - Smarak
... View more
12-08-2020
03:23 AM
Hello @kvinod Thanks for using Cloudera Community. Your concern is HBase OldWALs on HDFS Path "/hbase/oldWALs" are occupying a lot of space. HBase Replication isn't being used & TTL is set to 1 Minute. The HMaster Trace Logs capture the CleanerChore with verbose logging, yet I wish to check if you have tried the following 2 options: 1. Restart the HMaster Service to confirm if any issues with CleanerChore, 2. The Parameter " hbase.replication" is set correctly to False via the Steps shared under Section [1]. 3. If the "/hbase/replication" has any entries. If no Replication is utilised (HBase Replication or Lily Indexer), Try removing the "/hbase/replication" ZNode & restart the HMaster Service. - Smarak [1] CM=> HBase=> Configuration=> Advanced=> HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml
... View more
12-02-2020
09:46 PM
1 Kudo
Hello @nanda_bigdata
Sharing the Solution to ensure the Post is marked Completed. From WAL Reader, We confirmed the Writes to the RegionServer WAL pertains to 1 ColumnFamily only, indicating the Writes are arriving to 1 ColumnFamily only. It was confirmed that the wrong Hbase Configuration was being used by the Application. After ensuring the correct Hbase Configuration was used by Application, the issue was Fixed.
- Smarak
... View more
11-28-2020
06:52 AM
Hello @ibr If you are referring to the Supported Database for CDP Private Cloud Base (Which is the SDX for Private Cloud Experience), the list of Supported Databases for CDP Private Cloud Base, wherein the Databases for Metadata, Authorisation is maintained is shared in the Link [1]. [1] https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdpdc-database-requirements.html
... View more
11-22-2020
09:28 PM
Hello @Manoj690 RegionServer is a Service & your team can add the RegionServer Service interactively using via Ambari (HDP) or Cloudera Manager (CDH or CDP). - Smarak
... View more
11-12-2020
10:51 AM
Hello @lenu If you have Replication enabled, WALs are likely to be persisted until the WALs are replicated. If you aren't using HBase Replication, Ensure there are no Peers (via "list_peers") & " hbase.replication" Property is false. If the oldWALs aren't removed, Enable TRACE Logging for the HBase Master Service, which would print the CleanerChore Thread removing or skipping any entries. - Smarak
... View more
11-12-2020
10:43 AM
Hello @ebythomaspanick It appears you are hitting HBASE-20616. If you have verified that no other Procedures are in RUNNABLE State (Except for Truncate & Enable for the concerned Table), Sidelining the MasterProcWALs & Clearing the Temp Directory "/apps/hbase/data/.tmp" would ensure the TruncateTableProcedure aren't retried. Stop the Masters (Active & Standby) during the concerned Step to avoid any issues. - Smarak
... View more
11-12-2020
10:39 AM
Hello @phongnamam To your Queries, (I) Split tries to take an approximate centre point of the RowKey for the Region being split. In this manner, Yes, Region Split is a Horizontal Split. By default, each Table would have 1 Regions. You mentioned, you have 3 Tables & 5 Regions. The same is feasible as each Table have 1 Region (Total 3 Regions) & the Hbase:Meta/Hbase:NameSpace Table have 1 Region each, making total 5 Regions. If a Small Table have 5 Regions, It's likely using Split Point. (II) When a RegionServer dies, the Regions hosted on the RegionServer are distributed across Live RegionServers & when the RegionServer comes back online, the Master assign Regions to the RegionServer based on Balancer. The Master Logs would show if Balancer is moving Regions as part of Balancing. (III) Ideally, Masters shouldn't flip if 1 RegionServer stops. Best approach is to check the Logs of both Masters to see for any issues. - Smarak
... View more
11-12-2020
10:31 AM
Hello @TR7_BRYLE It's an old Post, yet trying to answer the Query. If the eventual goal of splitting 1 Snapshot into 3 Snapshots is to Merge them, We would like to know your Use-Case & what your Team is trying to achieve. Splitting Snapshot isn't feasible, yet there may be other choices to achieve the Use-Case your Team is achieve, if your Team can share the same. - Smarak
... View more
11-12-2020
10:27 AM
Hello @singhvNt It's an Old Post, yet answering the Query for posterity. MajorCompaction doesn't perform any Merging of Regions. Either Region Normalizer or MergeRegion Command merges the Regions. - Smarak
... View more
11-12-2020
10:21 AM
Hello @shubs If you have fixed the issue, Kindly update the Post with the Solution. The Exception is coming from DFSInputStream with EOFException. Worth checking if the RegionServer having issues has any Zero Length WAL File under the WAL Directory. Or, Enable TRACE Logging on the concerned RegionServer to capture additional details on the concerned Exception. Or, Checking which WAL is being replicated & any FS issues with the blocks associated with the WAL File. - Smarak
... View more
11-12-2020
10:11 AM
Hello @SimL YARN Timeline Server v2 Reader uses HBase for Storage. The Log & HBCK Report shows the Table "prod.timelineservice.entity" has Regions in OPENING State, which would naturally cause any Client accessing the Table report RetriesExhaustedWithDetailsException. A Region is typically stuck in OPENING State, if there is any issues with WALEditReplay or any other reason, which would be clear from the RegionServer Logs where the Region is being opened. Unless the RegionServer Logs are available, any Comment is unlikely to be an accurate assessment of the issue your team is facing. Would recommend checking the RegionServer Logs wherein the Regions are in OPENING State & based on the cause, plan accordingly. - Smarak
... View more
11-12-2020
10:02 AM
Hello @qu1etstorm The Logs FileNotFoundException & YouAreDeadException are typically observed after the Cause, so worth checking the Logs before the concerned 2 Exception. TheYouAreDeadException is thrown by the Master when a RegionServer reports and is already being processed as Dead. This can happen when aRegionServer loses its session but didn't figure it yet. So, Best to check with respect to the RegionServer ZNode Session tracking in the Logs, while comparing the time at when the Master detected the RegionServer Ephemeral ZNode as being removed. Typically, JVMPause are the most likely cause. - Smarak
... View more
11-12-2020
09:51 AM
Hello @amit_ The Post is couple of months old, yet wish to know if you have fixed the issue. The Lease seems to have expired after ~30 Minutes as per your Configuration. However, the Log shows a processingTime of ~30/~60 Seconds for ~5MB of Scan Size, so the Scanner Lease expiring after ~30 Minutes looks strange & appears to be caused by either the Client going away somehow or the processingTime being excessively slow. If you have fixed the issue, Will appreciate your feedback on the Post. - Smarak
... View more
11-04-2020
11:36 PM
Thank You @Seaport for sharing the Solution for wider audience. In short, you followed the Link [1] to resolve the Protocol Header Issue as utilised by the User in Link [2]. - Smarak [1] https://github.com/python-happybase/happybase/issues/161 [2] https://community.cloudera.com/t5/Support-Questions/Sharing-how-to-solve-HUE-and-HBase-connect-problem-on-CDH-6/td-p/82030
... View more
11-02-2020
05:08 AM
Hello @jiaozz It's an Old Post, yet submitting a response to anyone encountering such issue. Yet, Kindly test the 2 approaches below: (I) At Hive Session Level, Execute " ADD FILE <HBase Configuration File> " (II) Add the HBase Config to "HIVE_AUX_JARS_PATH" Path at Hive Service Level. In short, the issue appears upon HBase Config being not correctly set, even though the 1st Create Table Hive SQL runs successfully yet fails upon any Table created referencing the Hive Table created in the 1st SQL. - Smarak
... View more
10-29-2020
04:36 AM
1 Kudo
Hello @kras Unfortunately, there is no Clean Way to remove such Secondary Regions. The concerned issue is tracked via HBASE-24720. 1 Quick Fix would be to create a Snapshot on say, Table_A (Which had RegionReplication reduced from 2 to 1), CloneSnapshot to a new Table (Table_B). Table_B would have 1 Region. - Smarak
... View more
10-29-2020
12:22 AM
Hello @ASIF123 Atlas Janus is an HBase Table, created with Column Family specification only. The DDL of the concerned Atlas Table is available in the HMaster UI & Master Logs, when Atlas Service is initialised for 1st time. However, Column Families are added eventually & each Row of the concerned Table has Different Column Qualifier. In short, there is no Static Definition of Atlas Janus Table. Any Phoenix Table or View mapping requires explicit specification of the Column Family & Column Qualifier. Henceforth, the Phoenix Table or View DDL will be specific to Customer's Env. Your team can review the Atlas Janus Table Output via Scan & confirm the Column Qualifier against each Column Family. Accordingly, the Phoenix Table or View can be created. - Smarak
... View more
10-28-2020
11:21 PM
Hello @rameshlk Maybe this Post by Josh Elser would help, wherein he discussed using PQS behind Load Balancer to achieve High Availability: https://community.cloudera.com/t5/Community-Articles/Deploying-the-Phoenix-Query-Server-in-production/tac-p/245110 - Smarak
... View more
10-28-2020
11:12 PM
Hello @kras I was able to reproduce the concerned behaviour as well. Checking internally on the Steps to safely remove the Secondary Regions. Shall Update the Post once a Solution is found. - Smarak
... View more
10-28-2020
10:14 PM
Hello @Manoj690 The best way would be to check the RegionServer Logs wherein the Region are stuck in OPENING State. The fact that Count of Region-In-Transition is reducing with time & Increasing again upon HBase Service Restart indicates the Regions are likely waiting on WAL Split & Edit Replay. With 3 RegionServer, 7K RIT indicates you must be having more than 7K Regions in Total. As such, the RegionServer are overloaded & WAL Split & Edit Replay would be Slow. There are RegionOpening Thread, which can be Increased yet the Ideal Solution would be to Reduce the RegionCount or Increase RegionServer Count to ~300-400 Region/RegionServer. - Smarak
... View more
10-28-2020
09:58 PM
Hi @kumarkeshav Your Observation is Correct. The Config isn't exposed via Ambari v2.7.x yet adding the same via Custom HBase-Site.XML is prohibited with the " This property is already defined in this configuration file" message. The same isn't an issue with Ambari v2.6.x, wherein the concerned Write Limit is exposed. I am not familiar with the Exact Location in Ambari DB, wherein the concerned Parameter is exposed to be manually over-written. The Simplest Solution is the one you are possibly using i.e. Edit the XML for the RegionServer directly & restart them manually to ensure they are picking up the concerned Updated Configuration. However, this isn't practical in a Large Env unless an Automation Tool is used. If I come across an Easier Fix, I shall update you in the Post. - Smarak
... View more
10-28-2020
08:36 PM
Hello @MG-1 The Symptoms appears to match HBASE-20616. In your Case, You should follow the below Steps: (I) Confirm the only RUNNABLE Procedures are TruncateTableProcedure (II) Stop the HMaster Services (Active & Standby). (III) Sideline the Contents of MasterProcWALs Directory (/hbase/MasterProcWALs). (IV) Sideline the Table Region Directories from " /hbase/.tmp/" Directory. (V) Start the HMaster Services. The above Process ensures the Master doesn't attempt the Truncate Table Procedures by sidelining the MasterProcWALs Directory Contents & Sideline the Table Region Directories from "/hbase/.tmp/" Directory, else the Table won't be allowed to be Created again. - Smarak
... View more
07-18-2018
09:53 AM
Review Link "https://doc.lucidworks.com/lucidworks-hdpsearch/2.6/Guide-Install-Ambari.html" Section "Advanced solr-metrics".
... View more
- « Previous
- Next »