Created 12-23-2016 03:26 PM
can someone please guide me on how to debug ? I tried two times but the thread just dies and no one contributes to it anymore . I really need to get this fixed as I am unable to proceed further, I will appreciate anyone sticking to this thread till its resolved .
The only clue I have is the following error in the Ambari server log file when I try to reinstall ranger KMS in my HDP2.5 setup
22 Dec 2016 17:12:34,798 INFO [ambari-client-thread-177498] AbstractResourceProvider:810 - Caught an exception while updating host components, retrying : org.apache.ambari.server.AmbariException: The 'krb5-conf' configuration is not available22 Dec 2016 17:12:35,049 INFO [ambari-client-thread-177498] AbstractResourceProvider:925 - Received a updateHostComponent request, clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us, request={ clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us, desiredState=INSTALLED, state=null, desiredStackId=null, staleConfig=null, adminState=null}22 Dec 2016 17:12:35,053 ERROR [ambari-client-thread-177498] KerberosHelperImpl:1938 - The 'krb5-conf' configuration is not available22 Dec 2016 17:12:35,053 INFO [ambari-client-thread-177498] AbstractResourceProvider:810 - Caught an exception while updating host components, retrying : org.apache.ambari.server.AmbariException: The 'krb5-conf' configuration is not available^C
Created 12-23-2016 06:32 PM
Could you please try this?
Stop ambari server
Take ambari server backup
Run the below command
update clusterconfigmapping set selected=1 where type_name='krb5-conf' and version_tag='version1480534635088';
Start ambari server and try installing Ranger KMS.
Created 12-23-2016 03:49 PM
I am assuming cluster is kerberised.
Could you please send me output of
select * from clusterconfigmapping where type_name='krb5-conf';
Created 12-23-2016 03:55 PM
ambari=> select * from clusterconfigmapping where type_name='krb5-conf'; cluster_id | type_name | version_tag | create_timestamp | selected | user_name ------------+-----------+----------------------+------------------+----------+----------- 2 | krb5-conf | version1480451713980 | 1480451714516 | 0 | admin 2 | krb5-conf | version1480457371499 | 1480457371908 | 0 | admin 2 | krb5-conf | version1480512641006 | 1480512641350 | 0 | admin 2 | krb5-conf | version1480514713561 | 1480514713480 | 0 | admin 2 | krb5-conf | version1480534242770 | 1480534242886 | 0 | admin 2 | krb5-conf | version1480534438613 | 1480534438721 | 0 | admin 2 | krb5-conf | version1480534635088 | 1480534635219 | 0 | admin (7 rows)
ambari=>
Created 12-23-2016 04:19 PM
here is the krb5.conf file I have , and all the TGT for HIVE n HBASE are working fine so I don't think there is any issue with krb5.conf file.
[root@hadoop1 ~]# find / -name krb5.conf /usr/hdp/2.5.0.0-1245/knox/templates/krb5.conf /usr/hdp/2.4.3.0-227-DELME/knox/templates/krb5.conf /etc/krb5.conf /var/lib/ambari-server/resources/scripts/krb5.conf [root@hadoop1 ~]# [root@hadoop1 ~]# more /etc/krb5.conf [libdefaults] renew_lifetime = 7d forwardable = true default_realm = abc.com ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] abc.com = { admin_server = hadoop1.abc.com kdc = hadoop1.abc.com } [root@hadoop1 ~]#
Created 12-23-2016 03:49 PM
Seems the krb5-conf is missing or corrupted.
Please try Manually create the kerberos-env and krb5-conf by issuing the Ambari REST API call explained below: PUT /api/v1/clusters/CLUSER_NAME
[ { "Clusters": { "desired_config": { "type": "krb5-conf", "tag": "version1234", "properties": { "domains":"", "manage_krb5_conf": "true", "conf_dir":"/etc", "content" : "[libdefaults]\n renew_lifetime = 7d\n forwardable= true\n default_realm = {{realm|upper()}}\n ticket_lifetime = 24h\n dns_lookup_realm = false\n dns_lookup_kdc = false\n #default_tgs_enctypes = {{encryption_types}}\n #default_tkt_enctypes ={{encryption_types}}\n\n{% if domains %}\n[domain_realm]\n{% for domain in domains.split(',') %}\n {{domain}} = {{realm|upper()}}\n{% endfor %}\n{%endif %}\n\n[logging]\n default = FILE:/var/log/krb5kdc.log\nadmin_server = FILE:/var/log/kadmind.log\n kdc = FILE:/var/log/krb5kdc.log\n\n[realms]\n {{realm}} = {\n admin_server = {{admin_server_host|default(kdc_host, True)}}\n kdc = {{kdc_host}}\n }\n\n{# Append additional realm declarations below #}\n" } } } }, { "Clusters": { "desired_config": { "type": "kerberos-env", "tag": "version1234", "properties": { "kdc_type": "mit-kdc", "manage_identities": "false", "install_packages": "true", "encryption_types": "aes des3-cbc-sha1 rc4 des-cbc-md5", "realm" : "EXAMPLE.COM", "kdc_host" : "hdc.host", "admin_server_host" : "kadmin.host", "executable_search_paths" : "/usr/bin, /usr/kerberos/bin, /usr/sbin, /usr/lib/mit/bin, /usr/lib/mit/sbin", "password_length": "20", "password_min_lowercase_letters": "1", "password_min_uppercase_letters": "1", "password_min_digits": "1", "password_min_punctuation": "1", "password_min_whitespace": "0", "service_check_principal_name" : "${cluster_name}-${short_date}", "case_insensitive_username_rules" : "false" } } } } ]
Note: manage_identities is set to false indicating that Ambari is to not interact with the KDC. This is because the customer did not want Ambari to destroy the principals in the KDC. Since Ambari was not managing the Kerberos identities, there was no need to fill in the correct data about the KDC. TIP When issuing the API call mentioned above, place the payload into a file and use curl like:
curl -H "X-Requested-By:ambari" -u admin:admin -i -X PUT -d @./payload.json http://AMBARI_SEVER:8080/api/v1/clusters/CLUSTER_NAME
Created 12-23-2016 04:48 PM
if my krb5.conf file is corrupt how can I get valid TGT ? -bash-4.1$ kinit sami Password for sami@abc.com: -bash-4.1$ klist Ticket cache: FILE:/tmp/krb5cc_600 Default principal: sami@abc.com Valid starting Expires Service principal 12/23/16 11:47:19 12/24/16 11:47:19 krbtgt/abc.com@abc.com renew until 12/23/16 11:47:19 -bash-4.1$
Created 12-26-2016 02:03 PM
Hi @Sami Ahmad, It isn't the krb5.conf file that is corrupt but more the information that Ambari has in the database to manage your krb5.conf file. From what I am seeing above there isn't a configuration version selected and therefore Ambari is unable to find the configuration data. In my cluster I have a version selected for each which should be the last version. Here is what mine looks like. Notice the latest selected versions.
ambari=> select * from clusterconfigmapping where type_name = 'krb5-conf' or type_name = 'kerberos-env' order by version_tag desc; cluster_id | type_name | version_tag | create_timestamp | selected | user_name ------------+--------------+----------------------+------------------+----------+----------- 2 | krb5-conf | version1478018911089 | 1478018910394 | 1 | admin 2 | kerberos-env | version1478018911089 | 1478018910391 | 1 | admin 2 | kerberos-env | version1477959455789 | 1477959455113 | 0 | admin 2 | krb5-conf | version1477959455789 | 1477959455120 | 0 | admin 2 | kerberos-env | version1477959390268 | 1477959389823 | 0 | admin 2 | krb5-conf | version1477959390268 | 1477959389814 | 0 | admin 2 | krb5-conf | version1477956530144 | 1477956529438 | 0 | admin 2 | kerberos-env | version1477956530144 | 1477956529436 | 0 | admin 2 | krb5-conf | version1477687536774 | 1477687536111 | 0 | admin 2 | kerberos-env | version1477687536774 | 1477687536113 | 0 | admin 2 | krb5-conf | version1 | 1477680416621 | 0 | admin 2 | kerberos-env | version1 | 1477680416662 | 0 | admin (12 rows)This command will show me what Ambari thinks my latest version is and the content.
[root@chupa1 /]# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin get localhost myclustername krb5-conf USERID=admin PASSWORD=admin ########## Performing 'GET' on (Site:krb5-conf, Tag:version1478018911089) "properties" : { "conf_dir" : "/etc", "content" : "[libdefaults]\n renew_lifetime = 7d\n forwardable= true\n default_realm = {{realm|upper()}}\n ticket_lifetime = 48h\n dns_lookup_realm = false\n dns_lookup_kdc = false\n #default_tgs_enctypes = {{encryption_types}}\n #default_tkt_enctypes ={{encryption_types}}\n\n{% if domains %}\n[domain_realm]\n{% for domain in domains.split(',') %}\n {{domain}} = {{realm|upper()}}\n{% endfor %}\n{%endif %}\n\n[logging]\n default = FILE:/var/log/krb5kdc.log\nadmin_server = FILE:/var/log/kadmind.log\n kdc = FILE:/var/log/krb5kdc.log\n\n[realms]\n {{realm}} = {\n admin_server = {{admin_server_host|default(kdc_host, True)}}\n kdc = chupa1.openstacklocal\n }\n\n{# Append additional realm declarations below dav#}", "domains" : "", "manage_krb5_conf" : "true" }
Created 12-23-2016 06:32 PM
Could you please try this?
Stop ambari server
Take ambari server backup
Run the below command
update clusterconfigmapping set selected=1 where type_name='krb5-conf' and version_tag='version1480534635088';
Start ambari server and try installing Ranger KMS.
Created 12-23-2016 07:31 PM
you are a genius rgangappa .. . it got rid of the krb5-conf error. but now its complaining about the 'Kerberos-env' file ?
23 Dec 2016 14:29:52,975 INFO [ambari-client-thread-30] AbstractResourceProvider:810 - Caught an exception while updating host components, retrying : org.apache.ambari.server.AmbariException: The 'kerberos-env' configuration is not available 23 Dec 2016 14:29:53,226 INFO [ambari-client-thread-30] AbstractResourceProvider:925 - Received a updateHostComponent request, clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us, requ est={ clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us, desiredState=INSTALLED, state=null, desiredStackId=null, staleConfig=null, adminState=null} 23 Dec 2016 14:29:53,231 ERROR [ambari-client-thread-30] KerberosHelperImpl:1952 - The 'kerberos-env' configuration is not avai lable
Created 12-23-2016 08:21 PM
I fixed the Kerberos-env error the same way rgangappa suggested
update clusterconfigmapping set selected=1 where type_name='kerberos-env' and version_tag='version1480534635088';