Re: Pro and Cons using VMs for CDP

smdas · ‎03-08-2022

Hello @nikrahu We believe the Post by @ggangadharan answer your Queries. As such, We shall mark the Post as Resolved. If you have any concerns, Feel free to engage Cloudera Community via a Post. Thanks @ggangadharan for the detailed Examples !!! Regards, Smarak

smdas · ‎03-07-2022

Hello @OmarElSeihy Thanks for using Cloudera Community. Based on the Post, your Team is receiving a Warning [1]. By default, the Role uses "/tmp" for -XX:HeapDumpPath. Your Team need to change the Value set for -XX:HeapDumpPath to a Mount, wherein your Team have Sufficient Space. Alternatively, Your Team can work on reducing the FileSystem Usage on the concerned Mount Point. This Warning is an Alert for Cluster Admin & has no impact on any Service Usage in the Cluster. Sharing a Screenshot showing the Config Changes required, if your Team wish to modify the HeapDump Path & the Warning Threshold (Example shown for Solr Service as used Solr in the Post Tags): Regards, Smarak [1] This role's Heap Dump Directory is on a filesystem with less than 10.0 GiB of its space free. /tmp

smdas · ‎03-07-2022

Hello @Almolki Thanks for using Cloudera Community. Based on the Post, Your Team is experiencing [1] on Solr 8.4.1. The Issue being faced by your Team is a Known Limitation of Solr, wherein each Shard can host 2B Documents Maximum [2]. Your Team have following Choices: Use SplitShard to split the Shard with 2B Documents into 2 Daughter Shards. Refer [3] for using SplitShard API. If your Team have the Source Data, Create a Collection afresh with Higher Shards & Reindex the Source Data afresh. Kindly review & let us know if your Queries have been addressed. Regards, Smarak [1] Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 [2] https://issues.apache.org/jira/browse/SOLR-3504 [3] https://solr.apache.org/guide/8_4/shard-management.html#shard-management

smdas · ‎03-07-2022

Hello @ganeshkumarj Thanks for using Cloudera Community. Based on the Post, You are migrating from Cloudera Search (CDH 5.9.3) to Standalone Solr (Apache v4.10.3). As your Team mentioned, the Error points to Index (Copied manually) being on a Lucene Version higher than anticipated [1]. Your Team can confirm the LuceneVersion via "solrconfig.xml" for the Collection " sample_collection" on CDH. If LuceneVersion Match isn't feasible, ReIndexing is the Only Way forward. Yet, there are few things wherein our help in this Post would be limited: (I) CDH v5.9.3 is EoS since a long time. Internally, We have extremely limited Setup for checking further on your Team's concerns. (II) Your Team is implementing the Migration on Standalone Solr (Apache v4.10.3). Cloudera Product Offering package Solr into Search (In CDH) & Solr (In CDP). Unfortunately, We have limited input on any Open Source Implementation outside of Cloudera Product. Our Team would be happy to assist your Team to Migrate from CDH v5.9.3 to CDP, if required by your Team. We have Documentation (Which are Tested internally) to migrate from CDH Search to CDP Solr & your Team would get the Support assistance in any issues as well. Regards, Smarak [1] https://lucene.apache.org/core/7_1_0/core/org/apache/lucene/index/IndexFormatTooNewException.html

smdas · ‎03-04-2022

Greetings @Sayed016 Thanks for using Cloudera Community. HBase has Client Scan Timeout “hbase.client.scanner.timeout.period” (60 Seconds Default) along with Server RPC Timeout "hbase.rpc.timeout" (60 Seconds Default). I believe the Timeout being experienced by your Team is 1st one. Kindly set the 2 Parameters to 90 Seconds & ensure, Hive Client (Wherein your Team is accessing the Hive-On-Hbase Tables) is picking the Updated HBase Configurations. Regards, Smarak

smdas · ‎12-07-2021

Hello @AWlodarczyk Thanks for using Cloudera Community. The Link [1] covers the Minimum Requirement for both rows in their respective families. In other words, Azul JDK8 Family (8.56.0.21 & Above) & Azul JDK11 Family (11.50.19 & Above). They don't apply for Azul JDK13, Azul JDK15 & Azul JDK17 Family [2] for now. In [3], Your Team shall observe the JDK & the respective JDK Family Version being referred. Let us know if the above Post answer your queries. Regards, Smarak [1] https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-java-requirements.html? [2] https://www.azul.com/downloads/?package=jdk [3] https://supportmatrix.cloudera.com/

smdas · ‎12-07-2021

Hello @lbourgeois Apology for the delayed response & Thank You for the details. Internally, I could generate " HTTP ERROR 403 Forbidden", if I remove DEAdmin & DEUser from the Environment associated with the CDE Service for the User, for which the [User]:[Pass] is being passed. Once the above Privilege were added back to the User at Environment Level & " Synchronize Users" Operation Completed successfully, the Token was available (Wait for ~5 Minutes before retrying the Curl Command). Kindly review & let us know if the above Step works for you. Regards, Smarak

smdas · ‎12-07-2021

Hello @rootuser, Thanks for using Cloudera Community. Based on the Post, You are trying to use CopyTable to copy HBase Table(s) from 1 Cluster to another Cluster, wherein 1 Mapper is being observed. Please confirm if the Source Table has 1 Region only. Additionally, Confirm if CopyTable on a Table with >1 Regions (Say, 5 Regions) creates 1 Mapper or 5 Mappers. Also, Please state the HBase Version being used by your Team. Additionally, Share the Timeout being observed by your Team. As far as I recall, HBase uses 1 Mapper per Region. As such, It's likely the Source Table has 1 Region only. In such case, Increasing the Region Split by Pre-Split or Increasing the Timeout should help. Regards, Smarak

smdas · ‎12-01-2021

Hello @RafaelDiaz We hope the Post was helpful for you & marking the same as Resolved. If your Team continue to face the issue with HiveAccessControlException, Do update the Post & we can check accordingly. Regards, Smarak

smdas · ‎11-30-2021

Hello @lbourgeois Thanks for using Cloudera Community. Based on the Post, you are using to get CDE API Access Token & the Command just hangs. In short, You entered the Workload Password after using your Environment CDE Base URL followed by the KnoxToken Endpoint: ### curl -u <Your-Workload-User> <Your-CDE-Base-URL>/gateway/authtkn/knoxtoken/api/v1/token Kindly confirm if the behavior is Consistent across all CDE Services & all Users. Whether the DataLake (FreeIPA & IDBroker) is Up & Running. Additionally, Confirm the CDE Version being used. Regards, Smarak [1] https://docs.cloudera.com/data-engineering/cloud/api-access/topics/cde-api-get-access-token.html

smdas · ‎11-16-2021

Hello @xgxshtc Thanks for the Update. If you try to access the WAL File (For which the DFSOutputStream reports Premature EOF) via "hdfs dfs -cat/-head", Is the Command running successfully ? The FSCK Output can be modified to include " -openforwrite" to show the details of the 10 Files currently opened. Regards, Smarak

smdas · ‎11-15-2021

Hello @xgxshtc Thanks for the Update. By Sideline, We meant ensuring the HBase WAL Directory "/hbase/WALs" is Empty. Let us know if the below Steps helps: Stop HBase RegionServers, Sideline the WAL Directory Contents i.e. There shouldn't be any Directories with "/hbase/WALs", Restart HBase RegionServers. Additionally, You reinstalled the Cluster & yet observed the concerned Issue again. This likely indicates the HDFS State may be Unhealthy. Any Chance you can review the HDFS FSCK on the HBase WAL Directory [1] to confirm whether the Blocks associated with the HBase WAL Files are Healthy. Regards, Smarak [1] https://hadoop.apache.org/docs/r1.2.1/commands_manual.html#fsck

smdas · ‎11-13-2021

Hello @xgxshtc We observed you have posted the concerned ask in a New Post [1] as the concerned Post is ~4Years Old. While the Current Post is Unresolved, We shall wait on your Team's review on [1] before confirming the Solution on the Current Post as well. Regards, Smarak [1] https://community.cloudera.com/t5/Support-Questions/Hbase-regionserver-shutdown-after-few-hours/m-p/330070/highlight/false#M230589

smdas · ‎11-13-2021

Hello @xgxshtc Thanks for using Cloudera Community. Based on the Post, RegionServer is reporting EOFException | "Bad DataNode" while replaying WALs "/hbase/WALs". It appears the HDFS Blocks are having issues. To fix the Issue, Your Team can Sideline the Contents of the "/hbase/WALs" (Specific "/hbase/WALs/jfhbase03,60020,1636456250380") & restart the concerned RegionServer. If all RegionServers are Impacted, Sideline each of the RegionServer Directories from "/hbase/WALs". Note that the WALs hold the Edits not yet persisted to Disk & Sidelining the WAL Directories (1 WAL per RegionServer) may incur Data Loss. Additionally, Review the HDFS FSCK of HDFS files with WALs "/hbase/WALs" & fix any Corrupt/Missing Blocks. After ensuring the HDFS FSCK of WALs "/hbase/WALs" is Healthy, Your Team can restart the HBase RegionServers. Regards, Smarak

smdas · ‎11-11-2021

Hello @Faizan_Ali Thanks for using Cloudera Community. Based on the Post, ZooKeeper Service is failing to start on CDH v5.16. The Screenshot covers the "stdout" file. Kindly share the "stderr" file or the "Role Log" file for reviewing the issue with ZooKeeper Service. If your Team have Solved the Post, Kindly share the Issue & Steps performed to Mitigate the Issue. Your assistance would help our fellow Community Users. Regards, Smarak

smdas · ‎11-11-2021

Hello @drgenious Thanks for using Cloudera Community. We hope the response by @balajip was helpful for your ask. Additionally, We wish to share a few details: Your Question points to "How To Make Query Faster". Ideally, Impala would use Parallelism for executing a Query in fragments across Executors. As such, the 1st review should be using Impala Query Profile of the SQL to identify the Time taken at each Phase of SQL Execution. Refer [1] & [2] for few Links around Impala Query Profile. Once the Phase taking the Most Time is identified, Fine-Tune accordingly. Simply increasing the Impala Executors Daemon or using a Dedicated Coordinator may not be helpful, unless the SQL's Slow Fragment(s) are identified. Kindly review & let us know if you have any further ask in the Post. Regards, Smarak [1] https://cloudera.ericlin.me/2018/09/impala-query-profile-explained-part-1/ [2] https://docs.cloudera.com/runtime/7.2.10/impala-reference/topics/impala-profile.html

smdas · ‎11-10-2021

Hello @yacine_ Thanks for sharing the Solution along with the Root Cause as well. We shall mark the Post as Solved for now to ensure fellow Community Users can use the Solution as well. Regards, Smarak

smdas · ‎11-10-2021

Hello @vishal6196 Thanks for using Cloudera Community. Based on the Post, the Phoenix Delete Command is reporting more records being deleted than the Select Count(*) returns. Thank You for sharing the HDP & Phoenix Version as well. Few Queries for your Team: Whether the Observation is made for all Tables or Selective Table(s), Explain Output of the Select & Delete SQL, Any Index of the Phoenix Table being deleted, If Performing MajorCompaction on the Table before Deleting (Just for Sanity Check) shows any Difference with respect to the Rows Deleted via Delete SQL. Regards, Smarak

smdas · ‎11-10-2021

Hello @RafaelDiaz Thanks for using Cloudera Community. Based on the Post, Create Table via Hue works yet fails in R with "dbExecute" Function. While I haven't tried using "dbExecute" Function in R, the Error shows HiveAccessControlException denying the User on "default/iris". You haven't stated the CDH/CDP Version being used yet I assume your Team is using CDP with Ranger. First, Kindly check if passing the DB Name "cld_ml_bi_eng" with Table Name "iris" in "dbExecute" Function works. Since I haven't used "dbExecute" Function in R, Let us know if passing DB Name with Table Name like <DBName.TableName> is feasible. Secondly, Kindly check in Ranger for the Privilege of User "RDIAZ" for "CREATE" Privilege. Try including the DB Scope to "*" & Confirm the Table being Created with the Database ("default" or "cld_ml_bi_eng"). Accordingly, We can proceed further. Kindly review & share the Outcome from the above 2 suggestion. If your Team have Fixed the Issue already, We would appreciate your Team sharing the details for fellow Community Users as well. Regards, Smarak

smdas · ‎11-01-2021

Hello @SteffenMangold Kindly let us know if the Post was Helpful for your Team. If Yes, Please mark the Post as Solved. If No, Do share the Outcome of the Command shared in our Post. Accordingly, We shall review & get back to you. Regards, Smarak

smdas · ‎11-01-2021

Hello @Ma_FF Kindly let us know if the Post was Helpful to Identify the RegionServer Thread spiking the CPU Usage of the RegionServer JVM. If Yes, Please mark the Post as Solved. Regards, Smarak

smdas · ‎11-01-2021

Hello @paresh Thanks for using Cloudera Community. Based on the Post, HBase Service is being impacted owing to the Trace shared by you in the Post with the Solutions attempted. While Complete Logs helps, It appears the Region " 1595e783b53d99cd5eef43b6debb2682" Replay via "recovered.wals" is being interrupted owing to EOFException. When a Region is being Opened, any Contents in the "recovered.wals" are Replayed by reading them & pushing to MemStore. Once the Edits are persisted from MemStore to an Hfile, the "recovered.wals" are Removed. With the Possibility of DataLoss, You may attempt to Stop HBase > Sideline the RecoveredEdits "hdfs://ha-cluster:8020/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals" & MasterProcWALs (To avoid any Replay of associated PID) > Start HBase > Verify the Status. Note the "DataLoss" is likely from removing the RecoveredEdits File not persisted yet. Additionally, You may use WALPlayer [1] for replaying the Contents of RecoveredEdits as well. Regards, Smarak [1] https://hbase.apache.org/book.html#walplayer

smdas · ‎10-23-2021

Hello @amit_hadoop Thanks for using the Cloudera Community. It appears you are using [1] Post for asking the same Question as well. We have replied to you on the concerned Post. To avoid Duplication, We shall mark the Post as Closed. Regards, Smarak [1] https://community.cloudera.com/t5/Support-Questions/org-apache-hadoop-hbase-ipc-ServerNotRunningYetException/m-p/328376#M230250

smdas · ‎10-23-2021

Hello @Malvin We are marking the Post as Solved as Region Server Grouping is the Only Way to Segregate the Region Assignment at Region Server Level. If your Team encounter any issues, Feel free to Update the Post & we shall get back to your Team accordingly. Thank You again for using Cloudera Community. Regards, Smarak

smdas · ‎10-23-2021

Hello @rahuledavalath We are marking the Post as Solved. If your Team have any further concerns, Please update the Post, if your Team face any issues with either of the Recommendation shared [Migrate the System Tables Or Create a Phoenix Table afresh on Top of HBase Table]. Thank You for using Cloudera Community. Regards, Smarak

smdas · ‎10-23-2021

Hello @Rjkoop As stated by @willx, Visibility Labels aren't supported & Will has shared the Link. As such, We are marking the Post as Solved. Having said that, Your Team may post any further concerns in the Post. We shall review & get back to your Team accordingly. Thanks for using Cloudera Community. Regards, Smarak

smdas · ‎10-23-2021

Hello @Ma_FF Thanks for using Cloudera Community. Based on the Post, 1 Region Server is using High CPU. As requested by @PrathapKumar, Review the same. Additionally, Your Team can perform the below: (I) When the Region Server JVM reports High CPU, Open "top" Command for the Region Server PID, (II) Use "Shift H" to open the Thread View of the PID. This would show the Threads within the Region Server JVM with CPU Usage, (III) Monitor the Thread View & Identify the Thread hitting the Max CPU Usage, (IV) Take Thread Dump | JStack of Region Server PID & Compare the Thread with the "top" Thread View consuming the Highest CPU. The above Process would allow you to identify the Thread contributing towards the CPU Usage. Compare the same with other Region Server & your Team can make a Conclusive Call to identify the reasoning for CPU Utilization. Howsoever Logs are reviewed, Narrowing the Focus of JVM review would assist in identifying the Cause. Review shared Link for additional reference. Kindly review & share your Observation in the Post. Regards, Smarak [1] https://www.infoworld.com/article/3336222/java-challengers-6-thread-behavior-in-the-jvm.html [2] https://blogs.manageengine.com/application-performance-2/appmanager/2011/02/09/identify-java-code-consuming-high-cpu-in-linux-linking-jvm-thread-and-linux-pid.html [3] https://blog.jamesdbloom.com/JVMInternals.html

smdas · ‎10-21-2021

Hello @Sam7 We hope the SQL shared by @BennyZ helped your Team meet the Use-Case. Showing the SQL Output from the SQL shared in a Sample Table: As such, We are marking the Post as Solved. Regards, Smarak

smdas · ‎10-21-2021

Hello @Sayed016 Thanks for the response. In short, your Team identified GC Pauses during the concerned period wherein Solr Service was losing Connectivity with ZooKeeper. Your Team increased the zkClientTimeout to 30 Seconds & now, the Leader is elected for the Collection. In short, the Solr Service was impacted owing to GC Pauses of Solr JVM & the same has been addressed by your Team. Thank You for sharing your experience on the Community, which would help our fellow Community Users as well. Let us know if we are good to close the Post as Solved as well. Regards, Smarak

smdas · ‎10-21-2021

Hello @Malvin Thanks for using Cloudera Community. Based on the Post, You wish to have RegionServers to hold different number of Regions depending on the Hardware Spec of the Host. Note that RegionServer JVM runs with a Heap allocation, which is expected to be Uniform across the RegionServer(s). You may consider using RegionServer Grouping [1] to ensure Regions from Tables with Large Region Count are mapped to few RegionServers. The Link offers Implementation & Use-Cases details as well. Regards, Smarak [1] https://hbase.apache.org/book.html#rsgroup

Online	Offline
Last Visited	‎06-28-2022 05:34 AM

Date Registered	‎01-16-2018 09:55 AM
Last Visited	‎06-28-2022 05:34 AM
Total Messages Posted	385
Total Kudos Received	23

Re: Pro and Cons using VMs for CDP

Re: hbase I get an error "Master is initializing"

Re: setting "maxClientCnxns" option in zookeeper

Re: 7.2.12 - Streams Messaging Light Duty Error to...

Re: Solr Swap Memory exceeding threshold

Re: Can we change default hive/hbase timestamp fro...

Re: This role's Heap Dump Directory is on a filesy...

Re: solr process keeps stopping

Re: Migrate Cloudera solr 4.10.3-cdh-5.9.3 rto Apa...

Re: HBase CallTimeoutException for Hive on HBase t...

Re: Supported versions of Azul Zulu

Re: Unable to get a Cloudera Data Engineering API ...

Re: Hbase CopyTable Command Options

Re: Error in CREATE table with dbexecute in r

Re: Unable to get a Cloudera Data Engineering API ...

Re: Hbase regionserver shutdown after few hours

Re: Hbase regionserver shutdown after few hours

Re: HBase RegionServer stop automatically

Re: Hbase regionserver shutdown after few hours

Re: Failed to execute command Start on service Zoo...

Re: Impala executors and coordinators

Re: CDP7 can't access Atlas Web UI

Re: phoenix delete command deleting more record s ...

Re: Error in CREATE table with dbexecute in r

Re: Open a new HBase region without Hbase restart

Re: HBase: Region Server with High CPU usage

Re: master.HMasterCommandLine: Master exiting

Re: Timeline Service V2.0 Reader not starting due ...

Re: HBase tuning for computers with different har...

Re: Cloning Phoenix Hbase table snapshot from a HD...

Re: Hbase labels table creation

Re: HBase: Region Server with High CPU usage

Re: SQL statement to capitalize the first letter

Re: SolrException: ClusterState says we are the l...

Re: HBase tuning for computers with different har...