question Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm in Archives of Support Questions (Read Only)

When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

enor — Tue, 15 Dec 2015 06:44:34 GMT

Using the same stack in dev we successfully Kerberized the cluster. Now in production the process is being blocked at testing the KDC. The service check user is being created with a lower case realm;

prod_hdp-121415@abc.def.com instead of prod_hdp-121415@ABC.DEF.COM

When Ambari tries to kinit with this principal it fails;

resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/kerberos_service_check_cc_25fc6ba87c6a6872eb2c3b3167344f73 -kt /etc/security/keytabs/kerberos.service_check.121415.keytab hdp_prod-121415@abc.def.com' returned 1. kinit: Cannot find KDC for requested realm while getting initial credentials

To rule out a problem with the prod admin account and the prod container, the dev container and dev admin account were used in the production Ambari with the same result. So this is a configuration problem specific to the production environment.

Where should I look for something that would be lower casing the realm? It is correct in Ambari and the generated krb5.conf file.

The krb5.conf file;

[libdefaults]

renew_lifetime = 7d

forwardable = true

default_realm = ABC.DEF.COM

ticket_lifetime = 24h

dns_lookup_realm = false

dns_lookup_kdc = false

#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[domain_realm]

abc.def.com = ABC.DEF.COM

.abc.def.com = ABC.DEF.COM

[logging]

default = FILE:/var/log/krb5kdc.log

admin_server = FILE:/var/log/kadmind.log

kdc = FILE:/var/log/krb5kdc.log

[realms]

ABC.DEF.COM =

{

admin_server = ldap.abc.def.com

kdc = ldap.abc.def.com

}

########## Performing 'GET' on (Site:krb5-conf, Tag:version1450197141581)
"properties" : {
"conf_dir" : "/etc",
"content" : "\n[libdefaults]\n renew_lifetime = 7d\n forwardable = true\n default_realm = {{realm}}\n ticket_lifetime = 24h\n dns_lookup_realm = false\n dns_lookup_kdc = false\n #default_tgs_enctypes = {{encryption_types}}\n #default_tkt_enctypes = {{encryption_types}}\n\n{% if domains %}\n[domain_realm]\n{% for domain in domains.split(',') %}\n {{domain}} = {{realm}}\n{% endfor %}\n{% endif %}\n\n[logging]\n default = FILE:/var/log/krb5kdc.log\n admin_server = FILE:/var/log/kadmind.log\n kdc = FILE:/var/log/krb5kdc.log\n\n[realms]\n {{realm}} = {\n admin_server = {{admin_server_host|default(kdc_host, True)}}\n kdc = {{kdc_host}}\n }\n\n{# Append additional realm declarations below #}",
"domains" : "abc.def.com,.abc.def.com",
"manage_krb5_conf" : "true"
}

########## Performing 'GET' on (Site:kerberos-env, Tag:version1450197141581)
"properties" : {
"ad_create_attributes_template" : "\n{\n \"objectClass\": [\"top\", \"person\", \"organizationalPerson\", \"user\"],\n \"cn\": \"$principal_name\",\n #if( $is_service )\n \"servicePrincipalName\": \"$principal_name\",\n #end\n \"userPrincipalName\": \"$normalized_principal\",\n \"unicodePwd\": \"$password\",\n \"accountExpires\": \"0\",\n \"userAccountControl\": \"66048\"\n}",
"admin_server_host" : "ldap.abc.def.com",
"case_insensitive_username_rules" : "false",
"container_dn" : "OU=Hadoop,OU=Users,DC=abc,DC=def,DC=com",
"encryption_types" : "aes des3-cbc-sha1 rc4 des-cbc-md5",
"executable_search_paths" : "/usr/bin, /usr/kerberos/bin, /usr/sbin, /usr/lib/mit/bin, /usr/lib/mit/sbin",
"install_packages" : "true",
"kdc_create_attributes" : "",
"kdc_host" : "ldap.abc.def.com",
"kdc_type" : "active-directory",
"ldap_url" : "ldaps://ldap.abc.def.com:636",
"manage_identities" : "true",
"password_length" : "20",
"password_min_digits" : "3",
"password_min_lowercase_letters" : "1",
"password_min_punctuation" : "1",
"password_min_uppercase_letters" : "1",
"password_min_whitespace" : "0",
"realm" : "ABC.DEF.COM",
"service_check_principal_name" : "${cluster_name}-${short_date}"
}

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

nsabharwal — Tue, 15 Dec 2015 08:16:23 GMT

@Erik Nor Is it only for one service? or all of it?

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

enor — Tue, 15 Dec 2015 10:32:08 GMT

It is before it even gets to the other services. It happens during the "Test Kerberos Client" step. I haven't attempted to ignore it and proceed.

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

jeff1 — Tue, 15 Dec 2015 22:11:27 GMT

Hi, Which version of Ambari are you using?

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

rlevas — Wed, 16 Dec 2015 00:02:02 GMT

Can you post the results of the following API call?

GET /api/v1/clusters/CLUSTER_NAME/configurations?type=kerberos-env&fields=properties/*

Note: you will need to change CLUSTER_NAME to the name of your cluster.

For example:

http://ambari-server-host:8080/api/v1/clusters/MyCluster/configurations?type=kerberos-env&fields=properties/*

Basically, I am interested in the realm property. So you can just post that if the other information may be sensitive.

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

enor — Wed, 16 Dec 2015 00:37:36 GMT

Ambari v2.1.2

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

enor — Wed, 16 Dec 2015 00:50:08 GMT

I added it to the question. Thanks!

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

rlevas — Wed, 16 Dec 2015 01:08:11 GMT

On the host where the service check was executed, can you find the command.JSON file that was use to execute the command and post the contents of the "commandParams" block?

To find the correct command.json file do:

grep KERBEROS_SERVICE_CHECK /var/lib/ambari-agent/data/command-*.json

One or more files may be listed... any will be fine.

The commandParams from my cluster looks like:

"commandParams": {
  ...
  "principal_name": "c1-121515@EXAMPLE.COM",
  ...
}

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

enor — Wed, 16 Dec 2015 01:44:35 GMT

"principal_name": "hdp_prod-121515@abc.def.com",

Re: When Kerberizing via Ambari against an AD the service check principal is getting a lower case realm

rlevas — Wed, 16 Dec 2015 02:50:46 GMT

It appears that the UI stores the realm in the user-specified Kerberos descriptor artifact and that at some point the lowercase form of the realm was specified. However when the case of the realm was corrected, the user-specified Kerberos descriptor artifact was not updated and therefore the test Kerberos identity was created incorrectly.

I think this would have corrected itself if the test was skipped, but rather than take a chance on that, it is best to back out of the Kerberos Wizard (to the first page of it) and then exit - this makes sure the Kerberos service is cleaned up properly. Then the user-specified Kerberos descriptor artifact needs to be deleted using

DELETE /api/v1/clusters/CLUSTER_NAME/artifacts/kerberos_descriptor

Note: replace CLUSTER_NAME with the name of the cluster.

For example:

curl -H "X-Requested-By:ambari" -u admin:admin -i -X DELETE http://AMBARI_SERVER:8080/api/v1/clusters/CLUSTER_NAME/artifacts/kerberos_descriptor

Then restart the Enable Kerberos Wizard, making sure to set the realm with the proper case.