Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ranger KMS install failing

avatar
Super Collaborator

can someone please guide me on how to debug ? I tried two times but the thread just dies and no one contributes to it anymore . I really need to get this fixed as I am unable to proceed further, I will appreciate anyone sticking to this thread till its resolved .

The only clue I have is the following error in the Ambari server log file when I try to reinstall ranger KMS in my HDP2.5 setup

22 Dec 2016 17:12:34,798  INFO [ambari-client-thread-177498] AbstractResourceProvider:810 - Caught an exception while updating host components, retrying : org.apache.ambari.server.AmbariException: The 'krb5-conf' configuration is not available22 Dec 2016 17:12:35,049  INFO [ambari-client-thread-177498] AbstractResourceProvider:925 - Received a updateHostComponent request, clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us, request={ clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us, desiredState=INSTALLED, state=null, desiredStackId=null, staleConfig=null, adminState=null}22 Dec 2016 17:12:35,053 ERROR [ambari-client-thread-177498] KerberosHelperImpl:1938 - The 'krb5-conf' configuration is not available22 Dec 2016 17:12:35,053  INFO [ambari-client-thread-177498] AbstractResourceProvider:810 - Caught an exception while updating host components, retrying : org.apache.ambari.server.AmbariException: The 'krb5-conf' configuration is not available^C
1 ACCEPTED SOLUTION

avatar
Expert Contributor

@Sami Ahmad

Could you please try this?

Stop ambari server

Take ambari server backup

Run the below command

update clusterconfigmapping set selected=1 where type_name='krb5-conf' and version_tag='version1480534635088';

Start ambari server and try installing Ranger KMS.

View solution in original post

9 REPLIES 9

avatar
Expert Contributor
@Sami Ahmad

I am assuming cluster is kerberised.

Could you please send me output of

select * from clusterconfigmapping where type_name='krb5-conf';

avatar
Super Collaborator

ambari=> select * from clusterconfigmapping where type_name='krb5-conf'; cluster_id | type_name | version_tag | create_timestamp | selected | user_name ------------+-----------+----------------------+------------------+----------+----------- 2 | krb5-conf | version1480451713980 | 1480451714516 | 0 | admin 2 | krb5-conf | version1480457371499 | 1480457371908 | 0 | admin 2 | krb5-conf | version1480512641006 | 1480512641350 | 0 | admin 2 | krb5-conf | version1480514713561 | 1480514713480 | 0 | admin 2 | krb5-conf | version1480534242770 | 1480534242886 | 0 | admin 2 | krb5-conf | version1480534438613 | 1480534438721 | 0 | admin 2 | krb5-conf | version1480534635088 | 1480534635219 | 0 | admin (7 rows)

ambari=>

avatar
Super Collaborator

here is the krb5.conf file I have , and all the TGT for HIVE n HBASE are working fine so I don't think there is any issue with krb5.conf file.

[root@hadoop1 ~]# find / -name krb5.conf
/usr/hdp/2.5.0.0-1245/knox/templates/krb5.conf
/usr/hdp/2.4.3.0-227-DELME/knox/templates/krb5.conf
/etc/krb5.conf
/var/lib/ambari-server/resources/scripts/krb5.conf
[root@hadoop1 ~]#
[root@hadoop1 ~]# more /etc/krb5.conf
[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = abc.com
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  abc.com = {
    admin_server = hadoop1.abc.com
    kdc = hadoop1.abc.com
  }
[root@hadoop1 ~]#

avatar
Super Guru
@Sami Ahmad

Seems the krb5-conf is missing or corrupted.

Please try Manually create the kerberos-env and krb5-conf by issuing the Ambari REST API call explained below: PUT /api/v1/clusters/CLUSER_NAME

[
  {
    "Clusters": {
      "desired_config": {
        "type": "krb5-conf",
        "tag": "version1234",
        "properties": {
          "domains":"",
          "manage_krb5_conf": "true",
          "conf_dir":"/etc",
          "content" : "[libdefaults]\n  renew_lifetime = 7d\n  forwardable= true\n  default_realm = {{realm|upper()}}\n  ticket_lifetime = 24h\n  dns_lookup_realm = false\n  dns_lookup_kdc = false\n  #default_tgs_enctypes = {{encryption_types}}\n  #default_tkt_enctypes ={{encryption_types}}\n\n{% if domains %}\n[domain_realm]\n{% for domain in domains.split(',') %}\n  {{domain}} = {{realm|upper()}}\n{% endfor %}\n{%endif %}\n\n[logging]\n  default = FILE:/var/log/krb5kdc.log\nadmin_server = FILE:/var/log/kadmind.log\n  kdc = FILE:/var/log/krb5kdc.log\n\n[realms]\n  {{realm}} = {\n    admin_server = {{admin_server_host|default(kdc_host, True)}}\n    kdc = {{kdc_host}}\n }\n\n{# Append additional realm declarations below #}\n"
        }
      }
    }
  },
  {
    "Clusters": {
      "desired_config": {
        "type": "kerberos-env",
        "tag": "version1234",
        "properties": {
          "kdc_type": "mit-kdc",
          "manage_identities": "false",
          "install_packages": "true",
          "encryption_types": "aes des3-cbc-sha1 rc4 des-cbc-md5",
          "realm" : "EXAMPLE.COM",
          "kdc_host" : "hdc.host",
          "admin_server_host" : "kadmin.host",
          "executable_search_paths" : "/usr/bin, /usr/kerberos/bin, /usr/sbin, /usr/lib/mit/bin, /usr/lib/mit/sbin",
          "password_length": "20",
          "password_min_lowercase_letters": "1",
          "password_min_uppercase_letters": "1",
          "password_min_digits": "1",
          "password_min_punctuation": "1",
          "password_min_whitespace": "0",
          "service_check_principal_name" : "${cluster_name}-${short_date}",
          "case_insensitive_username_rules" : "false"
        }
      }
    }
  }
]
Note: 
manage_identities is set to false indicating that Ambari is to not interact with the KDC. This is because the customer did not want Ambari to destroy the principals in the KDC. Since Ambari was not managing the Kerberos identities, there was no need to fill in the correct data about the KDC.

TIP
When issuing the API call mentioned above, place the payload into a file and use curl like:

curl -H "X-Requested-By:ambari" -u admin:admin -i -X PUT -d @./payload.json http://AMBARI_SEVER:8080/api/v1/clusters/CLUSTER_NAME

avatar
Super Collaborator
if my krb5.conf file is corrupt how can I get valid TGT ?
-bash-4.1$ kinit sami
Password for sami@abc.com:
-bash-4.1$ klist
Ticket cache: FILE:/tmp/krb5cc_600
Default principal: sami@abc.com
Valid starting     Expires            Service principal
12/23/16 11:47:19  12/24/16 11:47:19  krbtgt/abc.com@abc.com
        renew until 12/23/16 11:47:19
-bash-4.1$


avatar

Hi @Sami Ahmad, It isn't the krb5.conf file that is corrupt but more the information that Ambari has in the database to manage your krb5.conf file. From what I am seeing above there isn't a configuration version selected and therefore Ambari is unable to find the configuration data. In my cluster I have a version selected for each which should be the last version. Here is what mine looks like. Notice the latest selected versions.

ambari=> select * from clusterconfigmapping where type_name = 'krb5-conf' or type_name = 'kerberos-env' order by version_tag desc;

 cluster_id |  type_name   |     version_tag      | create_timestamp | selected | user_name 

------------+--------------+----------------------+------------------+----------+-----------

          2 | krb5-conf    | version1478018911089 |    1478018910394 |        1 | admin

          2 | kerberos-env | version1478018911089 |    1478018910391 |        1 | admin

          2 | kerberos-env | version1477959455789 |    1477959455113 |        0 | admin

          2 | krb5-conf    | version1477959455789 |    1477959455120 |        0 | admin

          2 | kerberos-env | version1477959390268 |    1477959389823 |        0 | admin

          2 | krb5-conf    | version1477959390268 |    1477959389814 |        0 | admin

          2 | krb5-conf    | version1477956530144 |    1477956529438 |        0 | admin

          2 | kerberos-env | version1477956530144 |    1477956529436 |        0 | admin

          2 | krb5-conf    | version1477687536774 |    1477687536111 |        0 | admin

          2 | kerberos-env | version1477687536774 |    1477687536113 |        0 | admin

          2 | krb5-conf    | version1             |    1477680416621 |        0 | admin

          2 | kerberos-env | version1             |    1477680416662 |        0 | admin

(12 rows)

This command will show me what Ambari thinks my latest version is and the content.
[root@chupa1 /]# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin get localhost myclustername krb5-conf
USERID=admin
PASSWORD=admin
########## Performing 'GET' on (Site:krb5-conf, Tag:version1478018911089)
"properties" : {
"conf_dir" : "/etc",
"content" : "[libdefaults]\n renew_lifetime = 7d\n forwardable= true\n default_realm = {{realm|upper()}}\n ticket_lifetime = 48h\n dns_lookup_realm = false\n dns_lookup_kdc = false\n #default_tgs_enctypes = {{encryption_types}}\n #default_tkt_enctypes ={{encryption_types}}\n\n{% if domains %}\n[domain_realm]\n{% for domain in domains.split(',') %}\n {{domain}} = {{realm|upper()}}\n{% endfor %}\n{%endif %}\n\n[logging]\n default = FILE:/var/log/krb5kdc.log\nadmin_server = FILE:/var/log/kadmind.log\n kdc = FILE:/var/log/krb5kdc.log\n\n[realms]\n {{realm}} = {\n admin_server = {{admin_server_host|default(kdc_host, True)}}\n kdc = chupa1.openstacklocal\n }\n\n{# Append additional realm declarations below dav#}",
"domains" : "",
"manage_krb5_conf" : "true"
}

avatar
Expert Contributor

@Sami Ahmad

Could you please try this?

Stop ambari server

Take ambari server backup

Run the below command

update clusterconfigmapping set selected=1 where type_name='krb5-conf' and version_tag='version1480534635088';

Start ambari server and try installing Ranger KMS.

avatar
Super Collaborator

you are a genius rgangappa .. . it got rid of the krb5-conf error. but now its complaining about the 'Kerberos-env' file ?

23 Dec 2016 14:29:52,975  INFO [ambari-client-thread-30] AbstractResourceProvider:810 - Caught an exception while updating host    components, retrying : org.apache.ambari.server.AmbariException: The 'kerberos-env' configuration is not available
23 Dec 2016 14:29:53,226  INFO [ambari-client-thread-30] AbstractResourceProvider:925 - Received a updateHostComponent request,    clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us, requ   est={ clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.tolls.dot.state.fl.us,    desiredState=INSTALLED, state=null, desiredStackId=null, staleConfig=null, adminState=null}
23 Dec 2016 14:29:53,231 ERROR [ambari-client-thread-30] KerberosHelperImpl:1952 - The 'kerberos-env' configuration is not avai   lable

avatar
Super Collaborator

I fixed the Kerberos-env error the same way rgangappa suggested

update clusterconfigmapping set selected=1 where type_name='kerberos-env' and version_tag='version1480534635088';