Hello @SDL    This is an Old Thread & I assume your Team have moved on, yet wish to Update this Post for future references. It was observed that such Overnight Restart were resetting the default CleanUp (24 Hours) set via [1] in SolrConfig.XML of the respective Solr Collection (Sample from Ranger_Audits Collection). This caused the CleanUp to be postponed on a daily basis & causes Document PileUp beyond their Expiration. If Customer are restarting the Service nightly, It's advisable to set the CleanUp from 24 Hours to a Lower Value (Like, 20 or 22 Hours).    Regards, Smarak   [1]  <processor class="solr.processor.DocExpirationUpdateProcessorFactory"> <int name="autoDeletePeriodSeconds">86400</int> <str name="ttlFieldName">_ttl_</str> <str name="expirationFieldName">_expire_at_</str> </processor> ... View more

Hello @utehrani    Thanks for engaging Cloudera Community. Based on the Post, Pods for CML on ECS are failing, which isn't being resolved by Restarting the Pod as well.    The Error [1] is generally observed for few Checks/Configs failing & frankly, the review of such an issue wouldn't be easy with Community.   Since your Team installed ECS & deploying CML, We believe it's quicker for your Team to engage Cloudera Support via a Support Case while sharing the Version (CDP Private Cloud Base Version, Private Cloud Data Service Version) along with the Pod Listing "kubectl get pods -A" & the Logs from the affected Pods.   Also, Whether your Team have deployed other Data Services like CDW (Cloudera Data Warehouse) & CDE (Cloudera Data Engineering) on the concerned ECS Setup yet. If No, We would like to confirm if their Deployment is working fine or facing similar concerns.    Regards, Smarak   [1]  Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND ... View more

Hello @Kings    Thanks for using Cloudera Community. Based on your Post, You wish to reuse the Hardware used for MapR to run Cloudera with the Data Storage Setup.    The Hardware/Java/Network/Database requirement for CDP Private Cloud v7.1.8 (Latest Release) is available via [1]. Your Team can review the same & confirm if the existing Hardware & Setup meets the requirement. Note that your Team needs to be install Cloudera Manager & allow Cloudera Manager to perform the Installation of various Services (Including HDFS) across the Cluster Hosts.    Whether a Complete Lift & Shift works depends on the existing Setup & compliance with [1]. For information, Cloudera offers a Public Cloud & SaaS offering as well. Additionally, Customer can use the existing Hardware to turn into a full-fledged Kubernetes Cluster via Cloudera Manager to run Kubernetes based Workload (CDE, CDW, CML). The reason for stating the same is your Team selected "CDE" i.e. Cloudera Data Engineering in the Labels.    We believe this is a right point to engage with our Cloudera's Sales Team & find the exact fit for your Team's requirement in Cloudera's Hybrid Cloud Offering.    Kindly review & let us know.    Regards, Smarak   [1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/installation/topics/cdpdc-hardware-requirements.html   ... View more

Hello @Khairul_Hasan    Thanks for engaging Cloudera Community. Based on the Post, Your Team is receiving [1] while running the Command [2] in CDP v7.1.7. The Same Command was running with CDP v7.1.3.    If you include Verbose for ClassLoading (Similar to  "-verbose:class"), Your Team would confirm the Jar from which the Method is loaded. If your Team can pass the Phoenix Client & Phoenix Server jar explicitly in the ClassPath (The Path would be similar to "/opt/cloudera/parcels/<PhoenixParcelDir>/lib/phoenix/<PhoenixClientJar>" & "/opt/cloudera/parcels/<PhoenixParcelDir>/lib/phoenix/<PhoenixServerJar>"), We expect the Error to be managed. Kindly review the same & share the Outcome.    Regards, Smarak   [1]  Can't find method newStub in org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService! [2]  java -cp /etc/hadoop/conf.cloudera.hdfs/ssl-client.xml:/etc/hbase/conf.cloudera.hbase/hbase-site.xml:/etc/hadoop/conf.cloudera.hdfs/core-site.xml:/etc/hadoop/conf.cloudera.hdfs/hdfs-site.xm:/data/scripts/LeaApp-1.0-SNAPSHOT.jar net.ba.lea.transformation.FileActions "/data/scripts/msc/IN/" "/tmp/nss_processing/" "/data/scripts/msc/reject/" "250" "LEA.DBM_CDR_FILE_HEAD" "NSS" "jdbc:phoenix:gzvlcdpnode01.ba.net:2181:/hbase:phoenix/gzvlcdpnode02@BA.NET:/etc/security/keytab/phoenix.keytab" ... View more

Hello @sgenchev    While I am not familiar with building Charts to answer the Query, I shall share a few CLI approaches: (I) HBCK: Running "hbase hbck --details" prints [1] detailing the Regions per RegionServer. (II) Using JMX: Say, the HMaster UI is "http://HMaster1:22002/master-status".  Replace "master-status" with "jmx", which prints the JSON Output. Additional filtering can be done to capture the Regions per RegionServers.    Regards, Smarak   [1] HBCK Output: RegionServer: RegionServer1,22101,1655811831192 number of regions: 1 RegionServer: RegionServer2,22101,1655811830393 number of regions: 1 RegionServer: RegionServer2,22101,1655811832307 number of regions: 1 ... View more

Hello @cprakash    Let us know if you have fixed the ConnectionRefused Error to Master1 on Port 16000. As @rki_ mentioned, Review the HMaster Logs to confirm the reasoning for ConnectionRefused. Few possible scenarios being Port 16000 is being used by any other Service Or, "master1" isn't correctly being mapped as per DNS Or, Port 16000 may be blocked. Based on the HMaster Logs, We may offer further review.    As such, If the Issue persists, Share the HMaster Logs covering the concerned Stacktrace as well. If the Issue has been resolved, We shall appreciate if you can share the details into the Cause & Fix applied.    Regards, Smarak ... View more

Hello @LakshmiSegu    We hope your query was addressed by Shehbaz's response. In Summary, (I) Ensure you Username has an IDBroker Mapping (Actions > Manage Access > IDBroker Mappings).  (II) Include the "spark.yarn.access.hadoopFileSystems" Parameter to point to the S3 Path [1].    Regards, Smarak   [1] https://docs.cloudera.com/runtime/7.2.15/developing-spark-applications/topics/spark-s3.html   ... View more

Hello @caisch    Thanks for using Cloudera Community. Based on your Post, you wish to confirm if TTL for HDFS can be set to 90Days & other Services' TTL is set to 14Days.   Since you selected Solr, Let me answer your Post & you can let me know if I understood the Post differently. In Solr, We have Collections which may have Data on HDFS or Local. For Example, RangerAudits Collection may have Data on HDFS & Atlas Collection may have Data on Local. At each Collection Level, the "solrconfig.xml" captures the TTL [1] via DocExpirationUpdateProcessorFactory Class. You can configure the TTL at each Collection Level in Solr & they would Cleanup the Underlying  Data, be it on HDFS or Local. Using the above Example of RangerAudits using HDFS Storage & Atlas using Local Storage, We can set RangerAudits to expire at 90 Days & Atlas to expire at 14 Days, which in turn would remove the Underlying  Data from HDFS & Local respectively for RangerAudits & Atlas.    Kindly review & let us know if the above answers your Post. If No, You may clarify & we shall get back to you accordingly.   Regards, Smarak   [1] https://solr.apache.org/docs/8_4_0/solr-core/org/apache/solr/update/processor/DocExpirationUpdateProcessorFactory.html   ... View more

Hello @HiThere    Thanks for using Cloudera Community. To your Query, Kindly refer [1] & [2] for Resource Requirement for CDP v7.1.7. Note that Documentation refers to Hardware requirement in terms of Resources (CPU, Memory, Network, Disk) as opposed to Physical Machines or Virtual Machines. As long as your Team meet the Hardware requirement for Storage & Compute, the Virtualized & Bare-Metal choices shouldn't matter.   Regards, Smarak   [1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-hardware-requirements.html [2] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/concepts/topics/cm-vpc-networking.html   ... View more

Hello @yagoaparecidoti    Thanks for using Cloudera Community. Based on the Post, You encountered "Master Is Initializing" & ended up fixing the same using Command found on Internet. You ended up with few Tables in Broken State, requiring you to delete & recreate them.   For documenting the Post, the Issue observed is tracked via [1]. In the HMaster Logs, We should see HBase:Meta & HBase:Namespace Region, which aren't being assigned. Sample tracing shared via [1]. Once the same happens, Using HBCK2 Jar to assign the Region being reported as Unassigned is required. The Command to be used via HBCK2 Jar to assign the Region is shared via [1] as well.    The HBCK2 Command has other Options, which (If performed without any oversight) may cause issues with HBase Table availability. The "Master Is Initializing" Error Fix as shared via [1] shouldn't cause any Table to be in Broken State & without knowing the explicit details into the Broken State, It's harder to confirm if Delete-Recreate was the ONLY Way Or, We could have restored the Table in other ways.   Having said that, the Post covers "Master Is Initializing" Error & the same has been addressed. As such, We shall mark the Post as Closed. Feel free to share your observation with the Tables in Broken State in a New Post, if you wish to engage for the Community observation & feedback.    Regards, Smarak   [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#master-startup-cannot-progress-in-holding-pattern-until-region-onlined   ... View more

Greetings @yagoaparecidoti    Thanks for using Cloudera Community. Based on the Post, you wish to confirm if the  "maxClientCnxns" Parameter set via CDM (Assuming you mean CM) for ZooKeeper Service is referred in any file associated.    You may refer to the Value of "maxClientCnxns" in "zoo.cfg" file associated with the ZooKeeper Process. In CDP/CDH (Managed by CM), You may refer to the same within the ZooKeeper Process Directory under "/var/run/cloudera-scm-agent/process/<ZooKeeper-Process-Directory>/zoo.cfg". The concerned file would be referred in the "ps" Output of the ZooKeeper Process as well.    Regards, Smarak ... View more