[CDH 5.8 Kerberos] Generate Missing Credentials Error

[#1]

Hi,

I am trying to enable kerberos in CDH 5.8 with os of ubuntu 14.04 

All are fine until I try to do the part of Generate Missing Credentials

Please find the error.

/usr/share/cmf/bin/gen_credentials.sh failed with exit code 127 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin:/usr/sbin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/sbin:/usr/sbin
+ export KRB5_CONFIG=/var/krb5kdc/krb5.conf
+ KRB5_CONFIG=/var/krb5kdc/krb5.conf
+ export KRB5_KDC_PROFILE=/var/krb5kdc/kdc.conf
+ KRB5_KDC_PROFILE=/var/krb5kdc/kdc.conf
+ export DEFKTNAME=FILE:/etc/krb5.keytab
+ DEFKTNAME=FILE:/etc/krb5.keytab
+ CMF_REALM=ACME.COM
+ KEYTAB_OUT=/tmp/cmf6659933762842902135.keytab
+ PRINC=kafka/sb-node1.example.com@ACME.COM
+ MAX_RENEW_LIFE=432000
+ KADMIN='kadmin -k -t /tmp/cmf622833402687775807.keytab -p cloudera-scm/admin@ACME.COM -r ACME.COM'
+ RENEW_ARG=
+ '[' 432000 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "432000 sec"'
+ '[' -z /var/krb5kdc/krb5.conf ']'
+ echo 'Using custom config path '\''/var/krb5kdc/krb5.conf'\'', contents below:'
+ cat /var/krb5kdc/krb5.conf
+ kadmin -k -t /tmp/cmf622833402687775807.keytab -p cloudera-scm/admin@ACME.COM -r ACME.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey kafka/sb-node1.example.com@ACME.COM'
kadmin: relocation error: /usr/lib/x86_64-linux-gnu/libkadm5clnt_mit.so.9: symbol k5_buf_len, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference

[#2]

Are you running a modified Ubuntu 14.04 with non-standard repositories?

The kadmin command (from Ubuntu's packages of krb5-kdc and krb5-admin-server) we are invoking as the default PATH provides us is failing at a lower level library incompatibility error.

What are your versions of these packages? They should ideally be "1.12+dfsg-2ubuntu5.2", is that matching with what you have installed?
And also, did you attempt an upgrade of any other packages (such as libc, kernel, etc.) from 14.04 to a later version?

[#3]

Thanks for your reply. The version of krb5-kdc and krb5-admin-server is exactly "1.12+dfsg-2ubuntu5.2". Also, I guess that I didn't upgrade my any packages to other versions.

So I guess the repositories is somehow changed. And I restore the repositories with the steps with this link http://askubuntu.com/questions/124017/how-do-i-restore-the-default-repositories. However, I still got the same error while generating Missing Credentials.

[#4]

Could you also post your version numbers for the below package names?

libkrb5support0 [ideal: 1.12+dfsg-2ubuntu5.2]
libc6 [ideal: 2.19-0ubuntu6.9]

[#5]

dpkg -s libkrb5support0 | grep Version

Version: 1.12+dfsg-2ubuntu5.2

dpkg -s libc6 | grep Version

Version: 2.19-0ubuntu6.9

it seems that the version is ideal 

[#6]

Thank you for checking and reporting back, it appears something is very strange about your Ubuntu environment.

In my test environment with a 14.04 VM install, these commands appear to work normally. Can you verify if they work the same in yours (below was executed on the KDC host, for my realm name of UBUNTU.VM):

~> kadmin.local
kadmin: addprinc test/admin
…
kadmin: xst -k test-admin.keytab -norandkey test/admin
kadmin: quit
~>
~> kadmin -k -t test-admin.keytab -p test/admin@UBUNTU.VM -r UBUNTU.VM
kadmin: listprincs
…
kadmin: quit
~>

I'm particularly interested to know if both kadmin.local and kadmin (with an admin keytab) commands work for you normally outside of Cloudera Manager which is merely also invoking a similar command via a shell script.

[#7]

Thank you for your reply.

Per your request, my environment works with no error with the test command you provided. It successfully adds the principle test/admin to keytab test-admin.keytab with kadmin.local. 

Also, It works fine in part of authenticating as principal test/admin@MyRealmName with keytab test-admin.keytab. 

Please let me know if you want to know what console log printed while running your test command  

[#8]

Thanks for testing it out. Can you retry the CM command again? Perhaps it was a transient problem during some library changes.

Also, if the command still fails with the same error, try the full actual command CM is trying to run too:

kadmin -k -t test-admin.keytab -p test/admin@ACME.COM -r ACME.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey kafka/sb-node1.example.com@ACME.COM'

[#9]

Thanks for your help. I have tried the full command of CM is trying to run and it perfectly generated the principle of kafka/sb-node1.example.com@ACME.COM. 

Also, I have to tell you that I am setting all the kerberos environment as a root user. When I try to print out $USER in the gen_credentials.sh, it shows the User name of "cloudera-scm". I don't know whether it is the cause of the problem. Hope it helps.

[#10]

What do you specifically mean by "setting all the kerberos environment" - do you mean you need to use some environment variables before invoking commands?

Yes CM invokes the script as the cloudera-scm user but that shouldn't matter in terms of running the command. Somehow the right libs are not being used when CM runs kadmin, but when you run it directly it does appear to load the right ones. Perhaps you can run ldd on the kadmin binary from the script and outside and try to compare them.

[#11]

Yes, It means that I will use some environment variables which are set in bashrc.

Also, by running ldd on kadmin from script and outside, I found that there is differences between their results.

Command line - root

linux-vdso.so.1 => (0x00007ffdb1bad000)
libkadm5clnt_mit.so.10 => /usr/local/lib/libkadm5clnt_mit.so.10 (0x00007fd13679f000)
libkrb5.so.3 => /usr/local/lib/libkrb5.so.3 (0x00007fd1364ce000)
libk5crypto.so.3 => /usr/local/lib/libk5crypto.so.3 (0x00007fd13629d000)
libcom_err.so.3 => /usr/local/lib/libcom_err.so.3 (0x00007fd136099000)
libkrb5support.so.0 => /usr/local/lib/libkrb5support.so.0 (0x00007fd135e8d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd135ab2000)
libgssrpc.so.4 => /usr/local/lib/libgssrpc.so.4 (0x00007fd135895000)
libgssapi_krb5.so.2 => /usr/local/lib/libgssapi_krb5.so.2 (0x00007fd13564c000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fd135430000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd13522c000)
/lib64/ld-linux-x86-64.so.2 (0x0000564c783cb000)

CM

linux-vdso.so.1 => (0x00007fff078ce000)
libss.so.2 => /lib/x86_64-linux-gnu/libss.so.2 (0x00007fa458c31000)
libkadm5clnt_mit.so.9 => /usr/lib/x86_64-linux-gnu/libkadm5clnt_mit.so.9 (0x00007fa458a1e000)
libkrb5.so.3 => /usr/local/lib/libkrb5.so.3 (0x00007fa45874c000)
libk5crypto.so.3 => /usr/local/lib/libk5crypto.so.3 (0x00007fa45851c000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fa458318000)
libkrb5support.so.0 => /usr/local/lib/libkrb5support.so.0 (0x00007fa45810b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa457d46000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa457b42000)
libgssrpc.so.4 => /usr/local/lib/libgssrpc.so.4 (0x00007fa457924000)
libgssapi_krb5.so.2 => /usr/local/lib/libgssapi_krb5.so.2 (0x00007fa4576db000)
libcom_err.so.3 => /usr/local/lib/libcom_err.so.3 (0x00007fa4574d7000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fa4572bb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa45709d000)
/lib64/ld-linux-x86-64.so.2 (0x000055aa9bce6000)'

[#12 · Accepted solution]

Thanks, could you also illustrate what you provide as env-vars, and why
they are required? The standard Kerberos setup documentation of Ubuntu does
not require any preset env-vars.

The ldd difference is what is driving the problem, and from the looks of it
there may be multiple kerberos libraries installed on the system, but am
uncertain how it has ended up that way. If you resolve the library trouble,
CM would be able to run the command normally.

[#13]

Thanks, you are right. I just discovered that there are two kadmin packages installed for unknown reason. Maybe it is because I changed the PATH variable once and installed the kadmin in other path where is different from the default path setting in CM.

I solved the problem with correcting the PATH variable and reinstalling the package. Once again, thank you for you help