Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

[CDH 5.8 Kerberos] Generate Missing Credentials Error

avatar
Explorer

Hi,

 

I am trying to enable kerberos in CDH 5.8 with os of ubuntu 14.04 

All are fine until I try to do the part of Generate Missing Credentials

Please find the error.

 

/usr/share/cmf/bin/gen_credentials.sh failed with exit code 127 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin:/usr/sbin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin:/usr/sbin
+ export KRB5_CONFIG=/var/krb5kdc/krb5.conf
+ KRB5_CONFIG=/var/krb5kdc/krb5.conf
+ export KRB5_KDC_PROFILE=/var/krb5kdc/kdc.conf
+ KRB5_KDC_PROFILE=/var/krb5kdc/kdc.conf
+ export DEFKTNAME=FILE:/etc/krb5.keytab
+ DEFKTNAME=FILE:/etc/krb5.keytab
+ CMF_REALM=ACME.COM
+ KEYTAB_OUT=/tmp/cmf6659933762842902135.keytab
+ PRINC=kafka/sb-node1.example.com@ACME.COM
+ MAX_RENEW_LIFE=432000
+ KADMIN='kadmin -k -t /tmp/cmf622833402687775807.keytab -p cloudera-scm/admin@ACME.COM -r ACME.COM'
+ RENEW_ARG=
+ '[' 432000 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "432000 sec"'
+ '[' -z /var/krb5kdc/krb5.conf ']'
+ echo 'Using custom config path '\''/var/krb5kdc/krb5.conf'\'', contents below:'
+ cat /var/krb5kdc/krb5.conf
+ kadmin -k -t /tmp/cmf622833402687775807.keytab -p cloudera-scm/admin@ACME.COM -r ACME.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey kafka/sb-node1.example.com@ACME.COM'
kadmin: relocation error: /usr/lib/x86_64-linux-gnu/libkadm5clnt_mit.so.9: symbol k5_buf_len, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference

 

1 ACCEPTED SOLUTION

avatar
Mentor
Thanks, could you also illustrate what you provide as env-vars, and why
they are required? The standard Kerberos setup documentation of Ubuntu does
not require any preset env-vars.

The ldd difference is what is driving the problem, and from the looks of it
there may be multiple kerberos libraries installed on the system, but am
uncertain how it has ended up that way. If you resolve the library trouble,
CM would be able to run the command normally.

View solution in original post

12 REPLIES 12

avatar
Mentor
Are you running a modified Ubuntu 14.04 with non-standard repositories?

The kadmin command (from Ubuntu's packages of krb5-kdc and krb5-admin-server) we are invoking as the default PATH provides us is failing at a lower level library incompatibility error.

What are your versions of these packages? They should ideally be "1.12+dfsg-2ubuntu5.2", is that matching with what you have installed?
And also, did you attempt an upgrade of any other packages (such as libc, kernel, etc.) from 14.04 to a later version?

avatar
Explorer

Thanks for your reply. The version of krb5-kdc and krb5-admin-server is exactly "1.12+dfsg-2ubuntu5.2". Also, I guess that I didn't upgrade my any packages to other versions.

 

So I guess the repositories is somehow changed. And I restore the repositories with the steps with this link http://askubuntu.com/questions/124017/how-do-i-restore-the-default-repositories. However, I still got the same error while generating Missing Credentials.

 

 

avatar
Mentor
Could you also post your version numbers for the below package names?

libkrb5support0 [ideal: 1.12+dfsg-2ubuntu5.2]
libc6 [ideal: 2.19-0ubuntu6.9]

avatar
Explorer

dpkg -s libkrb5support0 | grep Version

Version: 1.12+dfsg-2ubuntu5.2

 

dpkg -s libc6 | grep Version

Version: 2.19-0ubuntu6.9

 

it seems that the version is ideal 

avatar
Mentor

Thank you for checking and reporting back, it appears something is very strange about your Ubuntu environment.

 

In my test environment with a 14.04 VM install, these commands appear to work normally. Can you verify if they work the same in yours (below was executed on the KDC host, for my realm name of UBUNTU.VM):

~> kadmin.local
kadmin: addprinc test/admin

kadmin: xst -k test-admin.keytab -norandkey test/admin
kadmin: quit
~>
~> kadmin -k -t test-admin.keytab -p test/admin@UBUNTU.VM -r UBUNTU.VM
kadmin: listprincs

kadmin: quit
~>

I'm particularly interested to know if both kadmin.local and kadmin (with an admin keytab) commands work for you normally outside of Cloudera Manager which is merely also invoking a similar command via a shell script.

avatar
Explorer

Thank you for your reply.

 

Per your request, my environment works with no error with the test command you provided. It successfully adds the principle test/admin to keytab test-admin.keytab with kadmin.local. 

 

Also, It works fine in part of authenticating as principal test/admin@MyRealmName with keytab test-admin.keytab. 

 

Please let me know if you want to know what console log printed while running your test command  

 

 

avatar
Mentor
Thanks for testing it out. Can you retry the CM command again? Perhaps it was a transient problem during some library changes.

Also, if the command still fails with the same error, try the full actual command CM is trying to run too:

kadmin -k -t test-admin.keytab -p test/admin@ACME.COM -r ACME.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey kafka/sb-node1.example.com@ACME.COM'

avatar
Explorer

Thanks for your help. I have tried the full command of CM is trying to run and it perfectly generated the principle of kafka/sb-node1.example.com@ACME.COM. 

 

Also, I have to tell you that I am setting all the kerberos environment as a root user. When I try to print out $USER in the gen_credentials.sh, it shows the User name of "cloudera-scm". I don't know whether it is the cause of the problem. Hope it helps.

avatar
Mentor
What do you specifically mean by "setting all the kerberos environment" - do you mean you need to use some environment variables before invoking commands?

Yes CM invokes the script as the cloudera-scm user but that shouldn't matter in terms of running the command. Somehow the right libs are not being used when CM runs kadmin, but when you run it directly it does appear to load the right ones. Perhaps you can run ldd on the kadmin binary from the script and outside and try to compare them.