@kvinod    You can use MySQL for the component please follow    https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_mysql.html ... View more

@EricL  Sorry about the confusion, the location of /etc/security/keytabs/* is specific to HDP the equivalent of /var/run/cloudera-scm-agent/process   so try to map  so the command    $ klist -kt  /var/run/cloudera-scm-agent/process/*.keytab   shoule be valid  ... View more

@kal  I have written an article in response to a similar question in HCC before, you have a couple of things to do before this can work! My 2 cents advice you should have first configured the 3 clusters using single KDC, then added a second KDC's on each cluster then proceed to configure Cross_Realm Trust. In the process, you would have gained some knowledge on the implementation having said that you will need KDCs in a Master/Slave configuration to be able to propagate principal and Keytabs between the 2 KDC using krb5_prop this is another chapter on its own. I have already noticed also an error in your krb5.conf in the below part, first you have 3 KDC's which are not replicating.there are specific steps to enable replication between KDC's see the one highlighted in ORANGE the values on the left in [lower case ] should mirror on the right in [upper case] see valid example in BLUE   [domain_realm] .xxxxxxx.net = XXXXXXX.NET xxxxxxx.net = XXXXXXX.NET .xxxxxxx.com = XXXXXXX.NET xxxxxxx.com = XXXXXXX.NET .sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET sys.xxxxxx.net = SPECTRA.xxxxxx.NET --Valid-- [domain_realm] .xxxxxxx.net = XXXXXXX.NET xxxxxxx.net = XXXXXXX.NET .xxxxxxx.com = XXXXXXX.COM xxxxxxx.com = XXXXXXX.COM .spectra.xxxxxxx.net = SPECTRA.XXXXXXX.NET spectra.xxxxxx.net = SPECTRA.xxxxxx.NET You should also configure /etc/hosts file on all the 3 clusters to have IP--HOSTNAME--ALIAS and the files should be copied to all hosts in the cluster if DNS is not resolving. The example below depicts hosts in the 3 different network segments # Cluster 1 192.168.0.1 node1.SPECTRA.XXXXXXX.NET node1 192.168.0.2 node2.SPECTRA.XXXXXXX.NET node2 .... 192.168.0.3 node3.SPECTRA.XXXXXXX.NET node3 # Cluster 2 192.168.1.10 node01.XXXXXXX.NET node01 192.168.1.20 node02.XXXXXXX.NET node02 ......... 192.168.1.30 node03.XXXXXXX.NET node01 # Cluster 3 192.168.2.30 nodex.XXXXXXX.COM nodex 192.168.2.40 nodey.XXXXXXX.COM nodey ......... 192.168.2.50 nodez.XXXXXXX.COM nodez if you could breakdown your steps it would be easier to achieve as I reiterated in the beginning but its doable task. Please let me know ... View more

@kal    whats the output of the below snippet? # klist -kt /etc/security/keytabs/hive.service.keytab Could you also share your krb5.conf? Please garble in the important info but not the format 🙂 ... View more

@kvinod  Is it a production cluster or not? If its the latter is there a reason for not using some other database like Mysql/MariaDB/Oracle? In my implementations I tend to favour these 3 hence I have more hands-on in your case you could try out MariaDB/Mysql as they are open source  ... View more

@irfangk1  It would be good to clarify whether its a CDH or HDP cluster managed by Ambari or not and any relevant information.     ... View more

@kvinod  For sure you have a double entry in your db.mgmt.properties  one simple way is to split the files as I have done below  [------]    Create a back of the current file cp /etc/cloudera-scm-server/db.mgmt.properties  /etc/cloudera-scm-server/db.mgmt.properties.ORIG Then override the db.mgmt.properties with first the  ORANGE if nothing works then the BLUE   $ sudo vi /etc/cloudera-scm-server/db.mgmt.properties # The source of truth for these settings # is the Cloudera Manager databases and # changes made here will not be reflected # there automatically. # com.cloudera.cmf.ACTIVITYMONITOR.db.type=postgresql com.cloudera.cmf.ACTIVITYMONITOR.db.host=hostname:7432 com.cloudera.cmf.ACTIVITYMONITOR.db.name=amon com.cloudera.cmf.ACTIVITYMONITOR.db.user=amon com.cloudera.cmf.ACTIVITYMONITOR.db.password=4WB4R5yxnp com.cloudera.cmf.REPORTSMANAGER.db.type=postgresql com.cloudera.cmf.REPORTSMANAGER.db.host=hostname:7432 com.cloudera.cmf.REPORTSMANAGER.db.name=rman com.cloudera.cmf.REPORTSMANAGER.db.user=rman com.cloudera.cmf.REPORTSMANAGER.db.password=WceGeruLNG com.cloudera.cmf.NAVIGATOR.db.type=postgresql com.cloudera.cmf.NAVIGATOR.db.host=hostname:7432 com.cloudera.cmf.NAVIGATOR.db.name=nav com.cloudera.cmf.NAVIGATOR.db.user=nav com.cloudera.cmf.NAVIGATOR.db.password=D2tjw5xjoE com.cloudera.cmf.NAVIGATORMETASERVER.db.type=postgresql com.cloudera.cmf.NAVIGATORMETASERVER.db.host=hostname:7432 com.cloudera.cmf.NAVIGATORMETASERVER.db.name=navms com.cloudera.cmf.NAVIGATORMETASERVER.db.user=navms com.cloudera.cmf.NAVIGATORMETASERVER.db.password=elJRINTAth [--------------------------separator-----------------------------] # The source of truth for these settings # is the Cloudera Manager databases and # changes made here will not be reflected # there automatically. # com.cloudera.cmf.ACTIVITYMONITOR.db.type=postgresql com.cloudera.cmf.ACTIVITYMONITOR.db.host=hostname:7432 com.cloudera.cmf.ACTIVITYMONITOR.db.name=amon com.cloudera.cmf.ACTIVITYMONITOR.db.user=amon com.cloudera.cmf.ACTIVITYMONITOR.db.password=O31A60K5SN com.cloudera.cmf.REPORTSMANAGER.db.type=postgresql com.cloudera.cmf.REPORTSMANAGER.db.host=hostname:7432 com.cloudera.cmf.REPORTSMANAGER.db.name=rman com.cloudera.cmf.REPORTSMANAGER.db.user=rman com.cloudera.cmf.REPORTSMANAGER.db.password=BPPShP0O9k com.cloudera.cmf.NAVIGATOR.db.type=postgresql com.cloudera.cmf.NAVIGATOR.db.host=hostname:7432 com.cloudera.cmf.NAVIGATOR.db.name=nav com.cloudera.cmf.NAVIGATOR.db.user=nav com.cloudera.cmf.NAVIGATOR.db.password=QHYL7zUSQe com.cloudera.cmf.NAVIGATORMETASERVER.db.type=postgresql com.cloudera.cmf.NAVIGATORMETASERVER.db.host=hostname:7432 com.cloudera.cmf.NAVIGATORMETASERVER.db.name=navms com.cloudera.cmf.NAVIGATORMETASERVER.db.user=navms com.cloudera.cmf.NAVIGATORMETASERVER.db.password=elJRINTAth   After testing the 2 options depending on success or failure then we could get another option. Make sure you have also followed these steps link.  https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_ig_extrnl_pstgrs.html#cmig_topic_5_6_1 Please revert      ... View more

@kvinod  In such a case, you should attach the logs because that's the only source we can use to investigate and get tips of maybe what happened. How long have you cluster been running before going down? Did you ever purge the CMS database?  On large deployments, it's a good strategy to put the monitor roles on their own hosts and isolation. Now that you have deleted the CMS hence lost the data therein you can safely create a new backend database and point the new config to that instance. DB's to recreate Reports Manager  Activity Monitor Please revert   ... View more

@subhash_parise3    Can you replace these lines with the below blue line with the orange   export SPARK_HISTORY_OPTS='-Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal=HTTP/ip-0-0-0-0@HDPCLUSTER.LOCAL,kerberos.keytab=/etc/security/keytabs/spnego.service.keytab"   With the below text block   export SPARK_HISTORY_OPTS='-Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter. params="type=kerberos,kerberos.principal ={ {spnego_principal}},kerberos.keytab={ {spnego_keytab}}"'   Restart and let me know      ... View more