Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Getting error when add new service in Cluster after enable Kerberos

avatar
Champion

Hi

 

CDH 5.7.x

 

I used to add new services in our cluster using Cloudera Manager without any issue before enable Kerberos. We have installed/enabled Kerberos now and everything is good for the existing services

 

But I want to add new service (Solr) and getting the following error 

 

Start Solr: Failed to start service

Execute command Start this Solr Server on role Solr Server: Command failed to run because this role has invalid configuration. 
Review and correct its configuration. First error: Role is missing Kerberos keytab.
Please run the Generate Missing Credentials command on the
Kerberos Credentials tab of the Administration -> Security page

 

I hv tried to Generate Missing Credentials in Admin -> security page but it end up with following error

 

 

/usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ CMF_REALM=REALM.COM
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf6942980384105255302.keytab
+ PRINC=solr/<<my_ipaddress>>@REALM.COM
+ MAX_RENEW_LIFE=432000
+ KADMIN='kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM'
+ RENEW_ARG=
+ '[' 432000 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "432000 sec"'
+ '[' -z /var/run/cloudera-scm-server/krb5920427054266466413.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb5920427054266466413.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb5920427054266466413.conf
+ kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey solr/<<my_ipaddress>>@REALM.COM'
WARNING: no policy specified for solr/<<my_ipaddress>>@REALM.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "solr/<<my_ipaddress>>@REALM.COM".
+ '[' 432000 -gt 0 ']'
++ kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'getprinc -terse solr/<<my_ipaddress>>@REALM.COM'
++ tail -1
++ cut -f 12
get_principal: Operation requires ``get'' privilege while retrieving "solr/<<my_ipaddress>>@REALM.COM".
+ RENEW_LIFETIME='Authenticating as principal root/admin@REALM.COM with keytab /var/run/cloudera-scm-server/cmf2028852611455413307.keytab.'
+ '[' Authenticating as principal root/admin@REALM.COM with keytab /var/run/cloudera-scm-server/cmf2028852611455413307.keytab. -eq 0 ']'
/usr/share/cmf/bin/gen_credentials.sh: line 35: [: too many arguments
+ kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'xst -k /var/run/cloudera-scm-server/cmf6942980384105255302.keytab solr/<<my_ipaddress>>@REALM.COM'
kadmin: Operation requires ``change-password'' privilege while changing solr/<<my_ipaddress>>@REALM.COM's key
+ chmod 600 /var/run/cloudera-scm-server/cmf6942980384105255302.keytab
chmod: cannot access `/var/run/cloudera-scm-server/cmf6942980384105255302.keytab': No such file or directory

>>

 

So I've manually added "solr/<<my_ipaddress>>@REALM.COM" using kadmin.local and tried to import from Admin -> security page.. no luck

 

so now my questions are

1. Is there any prequest to add a new service in Kerberoized cluster?

2. I cannot simply press "Generate Missing Credentials in Admin -> security page" Becuase How does my cluster knows which service I am going to add... it can be Solr, or something else?? Still I tried but it says nothing to generate

 

Thanks

Kumar

 

1 ACCEPTED SOLUTION

avatar
Champion

FYI... Everything is fine with kadmin.local but kadmin is not working properly.. the same issue was raised by someone else in stackoverflow...

 

I just followed the instruction.. The issue has been fixed now

 

http://stackoverflow.com/questions/23779468/kerberos-kadmin-not-working-properly

 

View solution in original post

6 REPLIES 6

avatar
Master Guru

Cloudera Manager will create the necessary keytabs automatically when adding a service to a Kerberos-enabled cluster.

Based on your exception when attempting to manually generate the crednetials, the user you have configured as your Cloudera Manager Principal is not an admin (does not have 'get' privilege to create principals).

 

Please see the information here:

 

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_s3_cm_principal.html

 

Note that when using MIT KDC, admin access is defined in /var/kerberos/krb5kdc/kadm5.acl

See this for more info:

https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/kadm5_acl.html

 

To give any principal with "/admin" all privileges, you could use the following:

 

*/admin@REALM.COM *

 

After you have made sure you have a user created in the KDC (cloudera-scm/admin@REALM.COM for example), you can import those credentials as described here:

 

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_s4_kerb_wizard.html#concept_ann...

 

Lastly, hosts always need to have a valid fully-qualified Domain Name (FQDN).  When you redacted information in your principal, I see you mention:

 

"solr/<<my_ipaddress>>@REALM.COM"

 

All principals in your CDH cluster should have the format of "name/FQDN@REALM"

 

For instance:

solr/solrhost.example.com@EXAMPLE.COM

 

Make sure all of your cluster hosts resolve their FQDNs via forward and reverse DNS.

 

I hope that helps.

 

Ben

avatar
Champion

@bgooley Thanks for quick reply. Let me double check all the points that you have mentioned.

 

In the mean time, I am still not clear with one point....

 

I believe my /var/kerberos/krb5kdc/kadm5.acl and other configurations are fine, because As I mentioned already, all the existing services (HDFS, Hive, Impala, Oozie, Hue, etc) are working fine. If there is a problem with my configuration, I should get the same error for my the existing services right.. why should I get error for only new service?

 

The only difference between existing and new services are

1. Existing services are added before enable Kerberos (everything is ok)

2. Trying to add New services now after enable Kerberos

 

any idea?

avatar
Master Guru

No problem...

 

The attempt to manually generate credentials fails when the "kadmin" command fails:

 

kadmin -k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab -p root/admin@REALM.COM -r REALM.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey solr/<<my_ipaddress>>@REALM.COM'

 

add_principal: Operation requires ``add'' privilege while creating "solr/<<my_ipaddress>>@REALM.COM".

 

This means that the user that the principal that Cloudera Manager used to execute the kadmin command did not have privilege to add the principal.

 

You can try running the same kadmin command from the command line on the Cloudera Manager host to see if you get the same error.  If you do, make sure that the kadm5.acl is correct.

 

Also note that the generate credentials process must create the principals; it cannot import existing principals created in the KDC.

 

-Ben

avatar
Master Guru

Note on my last message:

 

The keytab file referenced in the kadmin command issued by Cloudera Manager is removed by CM after the generate credentials is run.  So, you'll need to remove the "-k -t /var/run/cloudera-scm-server/cmf2028852611455413307.keytab" part of the kadmin command so you can be promted for the password.

 

avatar
Champion

@bgooley You are correct, I am getting the privilege error when I use kadmin but it is working fine with kadmin.local... I understand Generate Missing Credential will use kadmin instead of kadmin.local, so this is causing the trouble.

 

[root@abc]# kadmin
Authenticating as principal root/admin@REALM.COM with password.
Password for root/admin@REALM.COM:
kadmin:  addprinc -maxrenewlife "432000 sec" -randkey -pw hadoop1 solr/<<my_IP>>@REALM.COM
WARNING: no policy specified for solr/<<my_IP>>@REALM.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "solr/<<my_IP>>@REALM.COM".c

[root@abc]# kadmin.local
kadmin.local:  addprinc -maxrenewlife "432000 sec" -randkey -pw hadoop1 solr/<<my_IP>>@REALM.COM
WARNING: no policy specified for solr/<<my_IP>>@REALM.COM; defaulting to no policy
Principal "solr/<<my_IP>>@REALM.COM" created.

I tried to Import the credential using CM -> Admin -> Security. It says success message but I list the Kerberos credential, the principal is still missing for only solr

 

Successfully imported KDC Account Manager credentials.

 

so I've deleted the principal that i've added manually using kadmin.local.....

How to fix the issue with kadmin? so that I can use Generate Missing Credential option

 

Here I've listed my configuration, do you think any change required on it?

cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@REALM.COM *
hive@REALM.COM *
hdfs@REALM.COM

### kadmin.local: listprincs HTTP/<<my_ipaddress>>@REALM.COM K/M@REALM.COM cloudera-scm/admin@REALM.COM hdfs/<<my_ipaddress>>@REALM.COM hdfs@REALM.COM hive/<<my_ipaddress>>@REALM.COM hue/<<my_ipaddress>>@REALM.COM impala/<<my_ipaddress>>@REALM.COM kadmin/admin@REALM.COM kadmin/changepw@REALM.COM kadmin/<<my_ipaddress>>@REALM.COM krbtgt/REALM.COM@REALM.COM mapred/<<my_ipaddress>>@REALM.COM oozie/<<my_ipaddress>>@REALM.COM root/admin@REALM.COM root@REALM.COM sentry/<<my_ipaddress>>@REALM.COM yarn/<<my_ipaddress>>@REALM.COM zookeeper/<<my_ipaddress>>@REALM.COM

 

I've confirmed that my Fully qualified Domain Name (FQDN) is correct with my configurations

 

Note: I am using admin login in Cloudera manager to generate new principal and root/admin@REALM in CLI to add new principal

 

 

avatar
Champion

FYI... Everything is fine with kadmin.local but kadmin is not working properly.. the same issue was raised by someone else in stackoverflow...

 

I just followed the instruction.. The issue has been fixed now

 

http://stackoverflow.com/questions/23779468/kerberos-kadmin-not-working-properly