Member since
01-16-2018
385
Posts
23
Kudos Received
47
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
170 | 04-14-2022 01:29 AM | |
562 | 04-14-2022 01:12 AM | |
159 | 04-14-2022 12:58 AM | |
305 | 03-29-2022 01:13 AM | |
346 | 03-29-2022 01:00 AM |
06-22-2022
12:15 AM
Hello @sgenchev While I am not familiar with building Charts to answer the Query, I shall share a few CLI approaches: (I) HBCK: Running "hbase hbck --details" prints [1] detailing the Regions per RegionServer. (II) Using JMX: Say, the HMaster UI is "http://HMaster1:22002/master-status". Replace "master-status" with "jmx", which prints the JSON Output. Additional filtering can be done to capture the Regions per RegionServers. Regards, Smarak [1] HBCK Output: RegionServer: RegionServer1,22101,1655811831192 number of regions: 1 RegionServer: RegionServer2,22101,1655811830393 number of regions: 1 RegionServer: RegionServer2,22101,1655811832307 number of regions: 1
... View more
06-21-2022
11:43 PM
Hello @cprakash Let us know if you have fixed the ConnectionRefused Error to Master1 on Port 16000. As @rki_ mentioned, Review the HMaster Logs to confirm the reasoning for ConnectionRefused. Few possible scenarios being Port 16000 is being used by any other Service Or, "master1" isn't correctly being mapped as per DNS Or, Port 16000 may be blocked. Based on the HMaster Logs, We may offer further review. As such, If the Issue persists, Share the HMaster Logs covering the concerned Stacktrace as well. If the Issue has been resolved, We shall appreciate if you can share the details into the Cause & Fix applied. Regards, Smarak
... View more
06-21-2022
11:38 PM
Hello @LakshmiSegu We hope your query was addressed by Shehbaz's response. In Summary, (I) Ensure you Username has an IDBroker Mapping (Actions > Manage Access > IDBroker Mappings). (II) Include the "spark.yarn.access.hadoopFileSystems" Parameter to point to the S3 Path [1]. Regards, Smarak [1] https://docs.cloudera.com/runtime/7.2.15/developing-spark-applications/topics/spark-s3.html
... View more
06-21-2022
11:17 PM
Hello @caisch Thanks for using Cloudera Community. Based on your Post, you wish to confirm if TTL for HDFS can be set to 90Days & other Services' TTL is set to 14Days. Since you selected Solr, Let me answer your Post & you can let me know if I understood the Post differently. In Solr, We have Collections which may have Data on HDFS or Local. For Example, RangerAudits Collection may have Data on HDFS & Atlas Collection may have Data on Local. At each Collection Level, the "solrconfig.xml" captures the TTL [1] via DocExpirationUpdateProcessorFactory Class. You can configure the TTL at each Collection Level in Solr & they would Cleanup the Underlying Data, be it on HDFS or Local. Using the above Example of RangerAudits using HDFS Storage & Atlas using Local Storage, We can set RangerAudits to expire at 90 Days & Atlas to expire at 14 Days, which in turn would remove the Underlying Data from HDFS & Local respectively for RangerAudits & Atlas. Kindly review & let us know if the above answers your Post. If No, You may clarify & we shall get back to you accordingly. Regards, Smarak [1] https://solr.apache.org/docs/8_4_0/solr-core/org/apache/solr/update/processor/DocExpirationUpdateProcessorFactory.html
... View more
04-19-2022
01:07 AM
Hello @SVK If your queries concerning Apache Airflow has been addressed, Feel free to mark the Post as Solved. If you have any further ask, Kindly share the same & we shall get back to you accordingly. Regards, Smarak
... View more
04-19-2022
01:05 AM
Hello @HiThere We hope your query concerning VMs & recommendations around Hardware is answered. We are marking the Post as Closed. If you have any further concerns, Feel free to post your ask & we shall answer your queries. Regards, Smarak
... View more
04-14-2022
01:29 AM
Hello @HiThere Thanks for using Cloudera Community. To your Query, Kindly refer [1] & [2] for Resource Requirement for CDP v7.1.7. Note that Documentation refers to Hardware requirement in terms of Resources (CPU, Memory, Network, Disk) as opposed to Physical Machines or Virtual Machines. As long as your Team meet the Hardware requirement for Storage & Compute, the Virtualized & Bare-Metal choices shouldn't matter. Regards, Smarak [1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-hardware-requirements.html [2] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/concepts/topics/cm-vpc-networking.html
... View more
04-14-2022
01:23 AM
Hello @SVK Thanks for using Cloudera Community. Based on your post, You wish to confirm if Airflow is Supported by Cloudera. Summarizing the responses shared by my Colleagues for you: (I) Airflow isn't Supported by Cloudera in Standalone Mode, (II) In CDP Public Cloud [1] & CDP Private Cloud [2], CDE (Cloudera Data Engineering) uses Airflow. Any issues encountered with CDE Airflow on CDP Public Cloud & CDP Private Cloud is Supported. (III) CDE allows External Airflow to be used as well. Yet, Supportability is restricted to Cloudera Airflow Providers only. Refer [3]. If your queries are addressed, Feel free to mark the Post as Solved. Regards, Smarak [1] https://docs.cloudera.com/data-engineering/cloud/orchestrate-workflows/topics/cde-airflow-dag-pipeline.html [2] https://docs.cloudera.com/data-engineering/1.3.4/orchestrate-workflows/topics/cde-airflow-dag-pipeline.html [3] https://docs.cloudera.com/data-engineering/cloud/orchestrate-workflows/topics/cde-airflow-provider.html?
... View more
04-14-2022
01:12 AM
Hello @yagoaparecidoti Thanks for using Cloudera Community. Based on the Post, You encountered "Master Is Initializing" & ended up fixing the same using Command found on Internet. You ended up with few Tables in Broken State, requiring you to delete & recreate them. For documenting the Post, the Issue observed is tracked via [1]. In the HMaster Logs, We should see HBase:Meta & HBase:Namespace Region, which aren't being assigned. Sample tracing shared via [1]. Once the same happens, Using HBCK2 Jar to assign the Region being reported as Unassigned is required. The Command to be used via HBCK2 Jar to assign the Region is shared via [1] as well. The HBCK2 Command has other Options, which (If performed without any oversight) may cause issues with HBase Table availability. The "Master Is Initializing" Error Fix as shared via [1] shouldn't cause any Table to be in Broken State & without knowing the explicit details into the Broken State, It's harder to confirm if Delete-Recreate was the ONLY Way Or, We could have restored the Table in other ways. Having said that, the Post covers "Master Is Initializing" Error & the same has been addressed. As such, We shall mark the Post as Closed. Feel free to share your observation with the Tables in Broken State in a New Post, if you wish to engage for the Community observation & feedback. Regards, Smarak [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#master-startup-cannot-progress-in-holding-pattern-until-region-onlined
... View more
04-14-2022
12:58 AM
Greetings @yagoaparecidoti Thanks for using Cloudera Community. Based on the Post, you wish to confirm if the "maxClientCnxns" Parameter set via CDM (Assuming you mean CM) for ZooKeeper Service is referred in any file associated. You may refer to the Value of "maxClientCnxns" in "zoo.cfg" file associated with the ZooKeeper Process. In CDP/CDH (Managed by CM), You may refer to the same within the ZooKeeper Process Directory under "/var/run/cloudera-scm-agent/process/<ZooKeeper-Process-Directory>/zoo.cfg". The concerned file would be referred in the "ps" Output of the ZooKeeper Process as well. Regards, Smarak
... View more
04-08-2022
01:06 AM
Greetings @wazzu62 We wish to check if you have reviewed @araujo ask for further checks on the concerned issue. If required, Change the Port for the ATS HBase from 17020 to any Value to see if the same helps, assuming the Port is configured to accept request. Regards, Smarak
... View more
04-08-2022
01:00 AM
Hello @Girija Thanks for using Cloudera Community. Since the Post is an Older one, Wish to confirm if you have resolved the Issue. If Yes, Kindly assist by sharing the Solution for wider community audience. Wish to check if your Team had the Kerberos Ticket before submitting the request. Regards, Smarak
... View more
04-08-2022
12:51 AM
Greetings @stephen_obrien Thanks for using Cloudera Community. We see your Team is working with our Support Team for the concerned issue. Based on the Support engagement, We shall update the Post accordingly. Regards, Smarak
... View more
04-08-2022
12:46 AM
Hello @MadhuNP Thanks for using Cloudera Community. We see your Team is working with our Support Team for the concerned issue. Based on the Support engagement, We shall update the Post accordingly. Regards, Smarak
... View more
04-08-2022
12:33 AM
Hello @Neil_1992 & @maykiwogno While we wait for our Nifi Guru @MattWho review, Wish to provide a bit of information on the Lucene Exception. It appears Nifi Provenance Repository uses Lucene for indexing & the AlreadyClosedException means the Lucene Core being accessed has been Closed already, owing to FileSystemException with "Too Many Open Files" for the one of the Core Content "/provenance_repo/provenance_repository/lucene-8-index-1647749380623/_vd_Lucene80_0.dvd". Once AlreadyClosedException is reported, Restarting the Lucene Service would ensure the Cores are initialized afresh. Wish to check if your Team have attempted to increase the OpenFileLimit of the User running the Nifi Process to manage the FileSystemException with "Too Many Open Files" & restart Nifi, which I assume would restart the Lucene Cores as well. Note that the above answer is provided from Lucene perspective as I am not a Nifi Expert. My only intention to get your team unblocked, if the issue is preventing any Nifi concerns. Regards, Smarak
... View more
04-08-2022
12:22 AM
Hello @ISC Thanks for using Cloudera Community. Based on the Post, You are experiencing an Error as shared while using Python with Spark. We shall need full trace of the Error along with which Operation cause the same (Even though the Operation is apparent in the trace shared). Along with the full trace, the Client used (Example: pySpark) & the CDP/CDH/HDP release used. The above details would help us review your Issue & proceed accordingly. Regards, Smarak
... View more
04-08-2022
12:17 AM
Hello @AzfarB We hope the above Post has helped answer your concerns & offered an Action Plan to further review. We are marking the Post as Resolved for now. For any concerns, Feel free to post your ask in a Post & we shall get back to you accordingly. Regards, Smarak
... View more
04-06-2022
11:51 PM
Thank You @RangaReddy for this detailed write-up. The level of detailing is awesome 👏
... View more
03-29-2022
01:38 AM
Hello @Suresh_lakavath Since we haven't heard from your side concerning the Post, We are marking the Post as Closed for now. Feel free to Update the Post based on your Team's observation from the Action Plan shared on 03/09. Regards, Smarak
... View more
03-29-2022
01:17 AM
Hello @Moawad Hope you are doing well. Kindly let us know if the Post on 03/20 documenting few Links from CDH v6.x helped your Team. Regards, Smarak
... View more
03-29-2022
01:13 AM
1 Kudo
Hello @dutras As the Issue has been resolved via Support Case, We are marking the Post as resolved. For reference, this Case required Repair & associated Steps, which are verbose to be documented in Community. For anyone facing such issues, Kindly submit a Support Case. Regards, Smarak
... View more
03-29-2022
01:00 AM
Hello @AzfarB Thanks for using Cloudera Community. Based on the Post, Your Team observed Solr-Infra JVM reporting WARNING for Swap Space more than 200MB being utilised. Restarting the Solr-Infra JVM ensured the WARNING went away. Note that Swapping isn't Bad in general & the same has been discussed in detail by Community in [1] & [2]. Plus, Deleting RangerAudits Documents won't affect the same as Solr uses JVM as documented [3]. Indexed Documents aren't persisted in Memory unless Cached, thereby ensuring Deletion won't fix the Swapping guaranteed. As your Screenshot shows, the Host itself is running short on Memory (~99% Utilised) & Overall Swap is ~80% at ~47GB, out of which Solr-Infra is contributing <1GB. As documented in the below Links, Your Team can focus on the Host Level Usage & Considering Increasing the Swap Threshold from 200MB to at least 10% of the Heap i.e. 2GB for a Warning. 01 additional point can be made as to why Solr-Infra Restart helped resolved the WARNING. This needs to be looked at from the Host perspective as to the amount of Memory freed & Whether the Overall Swap Usage reduced at Host Level after Solr-Infra Restart as opposed to Solr-Infra WARNING being suppressed only. Regards, Smarak [1] https://unix.stackexchange.com/questions/2658/why-use-swap-when-there-is-more-than-enough-free-space-in-ram [2] https://chrisdown.name/2018/01/02/in-defence-of-swap.html [3] https://blog.cloudera.com/apache-solr-memory-tuning-for-production/
... View more
03-20-2022
08:51 PM
Hello @Moawad Thanks for using Cloudera Community. Based on the Post, Your Team is having issues connecting Kudu via pySpark. Kindly confirm whether a Simple Example [1] as documented in CDH 6.2.x Guide works for your Team. Regards, Smarak [1] https://docs.cloudera.com/documentation/enterprise/6/6.2/topics/kudu_development.html [2] https://kudu.apache.org/docs/developing.html
... View more
03-17-2022
12:07 AM
Hello @Suresh_lakavath Hope you are doing well. We wish to follow up with you on this Post. Regards, Smarak
... View more
03-09-2022
01:27 AM
Greetings @melmoumni Thanks for using Cloudera Community. For now, No to HA for Catalogd. This is coming from the fact stated in [1], wherein the Catalogd Unavailability don't leads to DataLoss & they can be removed to be added on a New Host without any impact. Similarly, the Upstream JIRA [2] remains Unresolved. Again, I shall keep the Post available for others to provide their feedback, if they have tested the same. In [2], One User share their experience, which doesn't appear to be Successful. Regards, Smarak [1] https://impala.apache.org/docs/build/html/topics/impala_proxy.html [2] https://issues.apache.org/jira/browse/IMPALA-2702
... View more
03-09-2022
01:18 AM
Hello @Suresh_lakavath Thanks for using Cloudera Community. Your concern is even a Single Node fails your HBase Client request. Please note that your HBase Client request referring to " ip-10-145-250-154.us-west-2.compute.internal" (Along with other Region Servers) would fail. Any HBase Client request referencing other Region Servers outside "ip-10-145-250-154.us-west-2.compute.internal" would be Successful. Your Team's Action Plan should ideally be to identify the reasoning of the Failed Server List of "ip-10-145-250-154.us-west-2.compute.internal" (Which would require HMaster Logs & Region Server Logs covering the Timestamp of the "Failed Server List"). Regards, Smarak [1] org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ip-10-145-250-154.us-west-2.compute.internal/10.145.250.154:60020
... View more
03-08-2022
02:07 AM
Hello @Sayed016 We are marking the Post as Solved for now. If you encounter any issues after adding the 2 Parameters as shared, Feel free to share the same as well. Regards, Smarak
... View more
03-08-2022
01:47 AM
Hello @Prasansai Thanks for using Cloudera Community. Your Team can refer to @DianaTorres response. Additionally, Note that any Patch for any Log4j Vulnerability would only be shared via Support Case. If your HDP Stack is EoS, Your Team can always engage with Cloudera Account Team to further review. Regards, Smarak
... View more
03-08-2022
01:42 AM
Hello @grlzz Thanks for using Cloudera Community. In Summary, the Link shared by your Team [1] is the Way to ensure a Runtime has any Packages installed without the need to have the Packages installed upon CML Job/Model/Experiment Run. Please make a Note of [2] with respect to Limitations, yet from your Post, it appears your Team is already familiar with the same. We shall mark the Post as Solved. Thank You for your Participation with Cloudera Community. Regards, Smarak [1] https://docs.cloudera.com/machine-learning/cloud/runtimes/topics/ml-customized-runtimes.html [2] https://docs.cloudera.com/machine-learning/cloud/runtimes/topics/ml-custom-runtime-limitations.html
... View more
03-08-2022
01:37 AM
Hello @corestack We hope the Post by @Azhar_Shaikh pointing to Link [1] helps your Team as there has been no further response from your side. As such, We shall mark the Post as Resolved. Feel free to share any concerns with your Team's CDP Adoption via a Post in Community & We shall help your Team. Regards, Smarak [1] https://community.cloudera.com/t5/Community-Articles/How-to-configure-Single-Sign-On-SSO-for-CDP-Public-Cloud-the/ta-p/300222
... View more