Getting error when add new service in Cluster after enable Kerberos

[#1]

Hi

CDH 5.7.x

I used to add new services in our cluster using Cloudera Manager without any issue before enable Kerberos. We have installed/enabled Kerberos now and everything is good for the existing services

But I want to add new service (Solr) and getting the following error 

Start Solr: Failed to start service

Execute command Start this Solr Server on role Solr Server: Command failed to run because this role has invalid configuration. 
Review and correct its configuration. First error: Role is missing Kerberos keytab. 
Please run the Generate Missing Credentials command on the 
Kerberos Credentials tab of the Administration -> Security page

I hv tried to Generate Missing Credentials in Admin -> security page but it end up with following error

/usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ CMF_REALM=REALM.COM
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf6942980384105255302.keytab
+ PRINC=solr/<<my_ipaddress>>@REALM.COM
+ MAX_RENEW_LIFE=432000
+ KADMIN='kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM'
+ RENEW_ARG=
+ '[' 432000 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "432000 sec"'
+ '[' -z /var/run/cloudera-scm-server/krb5920427054266466413.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb5920427054266466413.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb5920427054266466413.conf
+ kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey solr/<<my_ipaddress>>@REALM.COM'
WARNING: no policy specified for solr/<<my_ipaddress>>@REALM.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "solr/<<my_ipaddress>>@REALM.COM".
+ '[' 432000 -gt 0 ']'
++ kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'getprinc -terse solr/<<my_ipaddress>>@REALM.COM'
++ tail -1
++ cut -f 12
get_principal: Operation requires ``get'' privilege while retrieving "solr/<<my_ipaddress>>@REALM.COM".
+ RENEW_LIFETIME='Authenticating as principal root/admin@REALM.COM with keytab /var/run/cloudera-scm-server/cmf2028852611455413307.keytab.'
+ '[' Authenticating as principal root/admin@REALM.COM with keytab /var/run/cloudera-scm-server/cmf2028852611455413307.keytab. -eq 0 ']'
/usr/share/cmf/bin/gen_credentials.sh: line 35: [: too many arguments
+ kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'xst -k /var/run/cloudera-scm-server/cmf6942980384105255302.keytab solr/<<my_ipaddress>>@REALM.COM'
kadmin: Operation requires ``change-password'' privilege while changing solr/<<my_ipaddress>>@REALM.COM's key
+ chmod 600 /var/run/cloudera-scm-server/cmf6942980384105255302.keytab
chmod: cannot access `/var/run/cloudera-scm-server/cmf6942980384105255302.keytab': No such file or directory

>>

So I've manually added "solr/<<my_ipaddress>>@REALM.COM" using kadmin.local and tried to import from Admin -> security page.. no luck

so now my questions are

1. Is there any prequest to add a new service in Kerberoized cluster?

2. I cannot simply press "Generate Missing Credentials in Admin -> security page" Becuase How does my cluster knows which service I am going to add... it can be Solr, or something else?? Still I tried but it says nothing to generate

Thanks

Kumar

[#2]

Cloudera Manager will create the necessary keytabs automatically when adding a service to a Kerberos-enabled cluster.

Based on your exception when attempting to manually generate the crednetials, the user you have configured as your Cloudera Manager Principal is not an admin (does not have 'get' privilege to create principals).

Please see the information here:

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_s3_cm_principal.html

Note that when using MIT KDC, admin access is defined in /var/kerberos/krb5kdc/kadm5.acl

See this for more info:

https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/kadm5_acl.html

To give any principal with "/admin" all privileges, you could use the following:

*/admin@REALM.COM *

After you have made sure you have a user created in the KDC (cloudera-scm/admin@REALM.COM for example), you can import those credentials as described here:

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_s4_kerb_wizard.html#concept_ann...

Lastly, hosts always need to have a valid fully-qualified Domain Name (FQDN).  When you redacted information in your principal, I see you mention:

"solr/<<my_ipaddress>>@REALM.COM"

All principals in your CDH cluster should have the format of "name/FQDN@REALM"

For instance:

solr/solrhost.example.com@EXAMPLE.COM

Make sure all of your cluster hosts resolve their FQDNs via forward and reverse DNS.

I hope that helps.

Ben

[#3]

@bgooley Thanks for quick reply. Let me double check all the points that you have mentioned.

In the mean time, I am still not clear with one point....

I believe my /var/kerberos/krb5kdc/kadm5.acl and other configurations are fine, because As I mentioned already, all the existing services (HDFS, Hive, Impala, Oozie, Hue, etc) are working fine. If there is a problem with my configuration, I should get the same error for my the existing services right.. why should I get error for only new service?

The only difference between existing and new services are

1. Existing services are added before enable Kerberos (everything is ok)

2. Trying to add New services now after enable Kerberos

any idea?

[#4]

No problem...

The attempt to manually generate credentials fails when the "kadmin" command fails:

kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey solr/<<my_ipaddress>>@REALM.COM'

add_principal: Operation requires ``add'' privilege while creating "solr/<<my_ipaddress>>@REALM.COM".

This means that the user that the principal that Cloudera Manager used to execute the kadmin command did not have privilege to add the principal.

You can try running the same kadmin command from the command line on the Cloudera Manager host to see if you get the same error.  If you do, make sure that the kadm5.acl is correct.

Also note that the generate credentials process must create the principals; it cannot import existing principals created in the KDC.

-Ben

[#5]

Note on my last message:

The keytab file referenced in the kadmin command issued by Cloudera Manager is removed by CM after the generate credentials is run.  So, you'll need to remove the "-k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab" part of the kadmin command so you can be promted for the password.

[#6]

@bgooley You are correct, I am getting the privilege error when I use kadmin but it is working fine with kadmin.local... I understand Generate Missing Credential will use kadmin instead of kadmin.local, so this is causing the trouble.

[root@abc]# kadmin
Authenticating as principal root/admin@REALM.COM with password.
Password for root/admin@REALM.COM:
kadmin:  addprinc -maxrenewlife "432000 sec" -randkey -pw hadoop1 solr/<<my_IP>>@REALM.COM
WARNING: no policy specified for solr/<<my_IP>>@REALM.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "solr/<<my_IP>>@REALM.COM".c

[root@abc]# kadmin.local
kadmin.local:  addprinc -maxrenewlife "432000 sec" -randkey -pw hadoop1 solr/<<my_IP>>@REALM.COM
WARNING: no policy specified for solr/<<my_IP>>@REALM.COM; defaulting to no policy
Principal "solr/<<my_IP>>@REALM.COM" created.

I tried to Import the credential using CM -> Admin -> Security. It says success message but I list the Kerberos credential, the principal is still missing for only solr

Successfully imported KDC Account Manager credentials.

so I've deleted the principal that i've added manually using kadmin.local.....

How to fix the issue with kadmin? so that I can use Generate Missing Credential option

Here I've listed my configuration, do you think any change required on it?

cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@REALM.COM *
hive@REALM.COM *
hdfs@REALM.COM

###
kadmin.local:  listprincs
HTTP/<<my_ipaddress>>@REALM.COM
K/M@REALM.COM
cloudera-scm/admin@REALM.COM
hdfs/<<my_ipaddress>>@REALM.COM
hdfs@REALM.COM
hive/<<my_ipaddress>>@REALM.COM
hue/<<my_ipaddress>>@REALM.COM
impala/<<my_ipaddress>>@REALM.COM
kadmin/admin@REALM.COM
kadmin/changepw@REALM.COM
kadmin/<<my_ipaddress>>@REALM.COM
krbtgt/REALM.COM@REALM.COM
mapred/<<my_ipaddress>>@REALM.COM
oozie/<<my_ipaddress>>@REALM.COM
root/admin@REALM.COM
root@REALM.COM
sentry/<<my_ipaddress>>@REALM.COM
yarn/<<my_ipaddress>>@REALM.COM
zookeeper/<<my_ipaddress>>@REALM.COM

I've confirmed that my Fully qualified Domain Name (FQDN) is correct with my configurations

Note: I am using admin login in Cloudera manager to generate new principal and root/admin@REALM in CLI to add new principal

[#7 · Accepted solution]

FYI... Everything is fine with kadmin.local but kadmin is not working properly.. the same issue was raised by someone else in stackoverflow...

I just followed the instruction.. The issue has been fixed now

http://stackoverflow.com/questions/23779468/kerberos-kadmin-not-working-properly