Member since
01-16-2018
593
Posts
38
Kudos Received
94
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
408 | 05-17-2023 10:41 PM | |
1449 | 04-03-2023 09:44 PM | |
416 | 04-03-2023 02:40 AM | |
529 | 03-10-2023 07:36 AM | |
618 | 03-10-2023 07:17 AM |
01-16-2023
01:57 AM
Hello @mingtian Hope you are doing well. We wish to follow-up with you & check if the DEBUG Logging assisted in confirming the reasoning for Balancer Algorithm deciding against Region-Movement. If Yes, Kindly let us know if your Q in the Post has been answered or any further Q remains. Regards, Smarak
... View more
01-16-2023
01:55 AM
Hello @pankshiv1809 Hope you are doing well. We wish to follow-up on the Post & confirm whether your Team was requesting information into Dynamic Allocation to allow Spark adjust resources based on Workload requirement. Regards, Smarak [1] Job Scheduling - Spark 3.3.1 Documentation (apache.org)
... View more
01-16-2023
01:54 AM
Hello @panb We hope your Query has been addressed by us & shall mark the Post as Resolved. In Summary, Your Team needs to meet the requirement as stated in [1], which doesn't differentiate in Processor Type & I believe your Team is referring to Hygon Dhyana Processor. Note that we have shared the Hardware requirement is shared for CDP v7.1.8 as CDH isn't recommended now owing to End-Of-Life. As a Best Practise, I shall suggest engaging with Cloudera Account Team associated with Customer to perform any due diligence with respect to Supportability & Best Practices prior to onboarding Use-Cases into any new Platform, wherein Supportability is doubted by your Team. Regards, Smarak [1] Hardware Requirements | CDP Private Cloud (cloudera.com)
... View more
01-13-2023
01:59 AM
Hello @pankshiv1809 Thanks for using Cloudera Community. Based on your Post, Assuming [1] would help i.e. Using Dynamic Allocation to allow Spark adjust resources based on Workload requirement. Regards, Smarak [1] Job Scheduling - Spark 3.3.1 Documentation (apache.org)
... View more
01-13-2023
01:56 AM
Hello @panb Thanks for using Cloudera Community. As far as I am aware, your Team needs to meet the requirement as stated in [1], which doesn't differentiate in Processor Type & I believe your Team is referring to Hygon Dhyana Processor. Note that we have shared the Hardware requirement is shared for CDP v7.1.8 as CDH isn't recommended now owing to End-Of-Life. Regards, Smarak [1] Hardware Requirements | CDP Private Cloud (cloudera.com)
... View more
01-13-2023
01:47 AM
Hello @mingtian Thanks for using Cloudera Community. Based on your Post, We would suggest enabling DEBUG Logging for HMaster (Via HMaster UI To Avoid Any Restart) & trigger the Balancer. Generally, Balancer Algorithm may be deciding against running any Region-Alignment owing to Cost Factor [1]. The HMaster Debug Log would print such Balancer information for your review, upon which the Params discussed in [1] can be tuned to force Balancer, yet the Default Params are generally persisted for most Use-Cases. Note that Balancer Job isn't to merely fit Equal Regions per RegionServer. Balancer consider various Cost as defined by [1] to proceed with Region-Alignment. Regards, Smarak [1] StochasticLoadBalancer (Apache HBase 3.0.0-alpha-4-SNAPSHOT API)
... View more
01-13-2023
01:35 AM
1 Kudo
Hello @Ryan_2002 Thanks for using Cloudera Community. To your Q, the Driver Cap is the Engine/Resource Profile & the Executor's Resource Usage is defined by the SparkSession or "spark-defaults.conf" file within the Project wherein the Workbench Session is being created. Your Team can review the Pods in the User's Namespace & see the same i.e. upon a Workbench Session Creation, an Engine Pod is started with "Limits" set toEngine/Resource Profile Settings. After SparkSession is initialised, additional Pods are generated within the User's Namespace based on the Execution's Configs passed via SparkSession or "spark-defaults.conf" file. You may configure the Executor's Configs as per your usage yet the same depends on the CML Workspace AutoScale Range & InstanceType. Say, an InstanceType supporting 8 vCPU & Executors requesting 8 vCPU won't work. Similarly, AutoScale Max of 5 yet requesting Executors collectively utilising the Resource Limit of 5 Nodes. Hope the above helps answer your Post's queries. If Yes, Kindly mark the Post as Solved. If No, Feel free to share your concerns & we shall address accordingly. Regards, Smarak
... View more
01-02-2023
10:03 PM
Hello @quangbilly79 Thanks for using Cloudera Community. Based on your Post, you may consider "Kafka Gateway" as the Client for Kafka, which are setup on the Hosts wherein the same is added as per Cloudera Manager "Assign Roles". A Client/Gateway is familiar with the Service (Kafka in this Case) & all Client/Service Configs are available for the Client/Gateway without any manual intervention. Any changes made to the Service or Client Configs is pushed to the Service/Client Configuration by Cloudera Manager. Imagine a Scenario wherein you wish to run "hdfs dfs -ls" on a HDFS FileSystem. Simply running the Command won't work unless the Host wherein the Command "hdfs dfs -ls" is being run knows the Setup (HDFS FileSystem, NameNode, Port, Protocol). Review [1] for an Example. Adding an HDFS Gateway ensures User doesn't need to manually configure a Client/Gateway with Cloudera Manager doing the needful. Similarly, Kafka Gateway operates. Else, Customer need to manually configure the Client/Gateway Setup. Hope the above answer your query concerning the Gateway Role. Regards, Smarak [1] https://www.ibm.com/docs/en/spectrum-scale-bda?topic=hdfs-clients-configuration
... View more
12-27-2022
10:30 PM
Hello @sachin_saju Thanks for using Cloudera Community. You have 2 ask in the Post: 1. How to configure different Storage Policies with Cold & Hot Data, 2. Applying different Compression Algorithm in 1 Column Family. For Q2, I believe the same isn't feasible i.e. Compression Algorithm can be set at CF level. Review [1] for the Compression Algorithm recommendation around Hot & Cold type data. For Q1, I assume you are referring to HDFS Storage Policy. If Yes, the same is configured uniformly i.e. I am not sure if we can apply different HDFS Storage Policy for different data within the same CF. In HBase, We generally recommend SSD [2] for WAL, else the HBase Data relies on HDFS Storage Policy used. Alternatively, Use BackUp-Restore [3] for having a "Cold" Version of Data, which can be restored as per requirement. Regards, Smarak [1] https://hbase.apache.org/book.html#data.block.encoding.types [2] https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/configuring-hbase/topics/hbase-configure-storage-policy-wal.html [3] https://hbase.apache.org/book.html#br.overview
... View more
12-27-2022
10:19 PM
1 Kudo
Hello @sachin_saju Thanks for using Cloudera Community. Your queries concerning the Read Path is discussed between a fellow Community User & myself in [1]. Kindly review the same & let us know if the same answer the queries around Read Path. In Summary, Read Path relies on a Merge of BlockCache & MemStore prior to returning the Output to the End-User, thereby avoiding any Inconsistent Read. Refer [2] for few Diagram around the same to help explain the Read Merge Path. Concerning Doubt # 3, Our Community User asked a similar Q in [3]. I haven't reviewed this Use-Case internally around Hit/Miss Ratio in the UI to answer the same. Henceforth, I shall let our fellow HBase Engineers to answer [3], which may answer your Q3 as well. Barring Q3, Let me know if your first 2 queries are addressed by [1] & [2]. Regards, Smarak [1] https://community.cloudera.com/t5/Support-Questions/Is-it-possible-for-inconsistent-read-in-Hbase-with-Memcache/m-p/359452#M238123 [2] https://nag-9-s.gitbook.io/hbase/hbase-architecture/hbase-read-merge [3] https://community.cloudera.com/t5/Support-Questions/Create-cache-miss-scenario-in-HBase-with-HDP-2-6-5/m-p/359795
... View more