Created on 01-27-2020 11:08 PM - last edited on 01-28-2020 03:57 AM by ask_bill_brooks
I'm having an issue with generating kerberos credentials.
Here's the stack trace from cloudera manager:
/usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ CMF_REALM=cloudera-scm/admin
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf538006877510569158.keytab
+ PRINC=hue/elhbdmanager1-test3.elhub.cloud@EXAMPLE.COM
+ MAX_RENEW_LIFE=432000
+ KADMIN='kadmin -k -t /var/run/cloudera-scm-server/cmf4481530940122639782.keytab -p cloudera-scm/admin -r cloudera-scm/admin'
+ RENEW_ARG=
+ '[' 432000 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "432000 sec"'
+ '[' -z /etc/krb5.conf ']'
+ echo 'Using custom config path '\''/etc/krb5.conf'\'', contents below:'
+ cat /etc/krb5.conf
+ kadmin -k -t /var/run/cloudera-scm-server/cmf4481530940122639782.keytab -p cloudera-scm/admin -r cloudera-scm/admin -q 'addprinc -maxrenewlife "432000 sec" -randkey hue/elhbdmanager1-test3.elhub.cloud@EXAMPLE.COM'
kadmin: Missing parameters in krb5.conf required for kadmin client while initializing kadmin interface
The problem is quite obvious: `CMF_REALM=cloudera-scm/admin` where it should be `CMF-REALM=EXAMPLE.COM`.
I'm not sure why it's not setting the correct realm though.
Can anyone help figure this out?
Created 01-29-2020 10:55 AM
Hi @sergei_P ,
What is your Cloudera Manager version? Is it same as CDH version? Did you use the wizard to enable Kerberos? If so, you can read this doc section "KRB5 configuration" for more information:
Manage krb5.conf through Cloudera Manager allows you to choose whether Cloudera Manager should deploy the krb5.conf on your cluster or not. If left unchecked, you must ensure that the krb5.conf is deployed on all hosts in the cluster, including the Cloudera Manager Server's host.
So depends if you choose above setting, you need to either check whether you configure the "Kerberos Security Realm" correctly from UI or you need to check whether /etc/krb5.conf are correctly configured on all hosts manually.
Thanks and hope this helps,
Li
Li Wang, Technical Solution Manager
Created 01-29-2020 10:20 PM
Hi @lwang , thanks for your input.
Cloudera manager version is 5.8.5, and CDH 5.4.4 (do they have to be the same version?)
No, I haven't used the wizard for Kerberos. And I've checked the krb5.conf file - it looks fine, but here it is just for the sake of having another pair of eyes on it 🙂
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = false
rdns = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
[appdefaults]
validate = false
[realms]
EXAMPLE.COM = {
kdc = bdmanager1-test2.example.com:88
admin_server = bdmanager1-test2.example.com:749
#default_domain = example.com
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
`krb5.conf` is deployed to all hosts, that's not the issue here.
Let me give an example. When I do "Import KDC admin credentials" from the Cloudera Manager UI and right after it's done I do "Generate missing credentials" - it works.
If I execute any commands in between, for example I do "Import cluster template" and that itself will call "Generate missing credentials" after deploying the parcels and applying host templates, then "Generate missing credentials" fails with the error that I specified above. If I manually again do "Import KDC admin credentials" and then "Generate missing credentials" - again it works.
The error comes from the script that is triggered by the "Generate missing credentials" command, located at `/usr/share/cmf/bin/gen_credentials.sh`, as it sets incorrect realm for `CMF_REALM` variable (as can be seen from the stack trace above), hence the error.
As a workaround I've explicitly set `CMF_REALM=EXAMPLE.COM` in that script, but that's not ideal so I am trying to get to the bottom of why this is happening.
Hope my ramblings make any sense 🙂
Created 01-30-2020 01:15 PM
@sergei_P I am not very sure about 5.4.4 as it's very old version an unsupported now [1]. But in the recent versions the script looks valid to me, as we have modified this parameter more precisely.
#!/usr/bin/env bash
# Copyright (c) 2011 Cloudera, Inc. All rights reserved.
set -e
set -x
# Explicitly add RHEL5/6, SLES11/12 locations to path
export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:$PATH
CMF_REALM=${CMF_PRINCIPAL##*\@}
KEYTAB_OUT=$1
PRINC=$2
MAX_RENEW_LIFE=$3
KADMIN="kadmin -k -t $CMF_KEYTAB_FILE -p $CMF_PRINCIPAL -r $CMF_REALM"
Notice this CMF_REALM=${CMF_PRINCIPAL##*\@} in code. You may try to tweak script in this way.
I would recommend to use some newer version of Cloudera Data Hub.
[1] https://www.cloudera.com/legal/policies/support-lifecycle-policy.html
Cheers
Created 01-30-2020 03:02 PM
Hi @sergei_P ,
It is ok to use CM5.8.5 with CDH5.4.4 because CM version is higher. The Cloudera Manager minor version must always be equal to or greater than the CDH minor version. Older versions of Cloudera Manager might not support features in newer versions of CDH. See this doc:
As for the issue you are experiencing, I wonder what is the value for this file?
/var/kerberos/krb5kdc/kadm5.acl
Try to use kadmin to list all the principals to confirm whether cloudera-scm/admin@EXAMPLE.COM is correctly set up.
Thanks,
Li
Li Wang, Technical Solution Manager
Created 01-31-2020 12:24 AM
The `var/kerberos/krb5kdc/kadm5.acl` file contents:
*/admin@EXAMPLE.COM *
cloudera-scm@EXAMPLE.COM * flume/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * hbase/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * hdfs/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * hive/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * httpfs/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * HTTP/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * hue/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * impala/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * mapred/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * oozie/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * solr/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * sqoop/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * yarn/*@EXAMPLE.COM
cloudera-scm@EXAMPLE.COM * zookeeper/*@EXAMPLE.COM
Here's the cloudera-scm/admin principal details:
kadmin.local: getprinc cloudera-scm/admin@EXAMPLE.COM
Principal: cloudera-scm/admin@EXAMPLE.COM
Expiration date: [never]
Last password change: Thu Jan 30 11:07:33 UTC 2020
Password expiration date: [none]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Thu Jan 30 11:07:33 UTC 2020 (root/admin@EXAMPLE.COM)
Last successful authentication: Thu Jan 30 11:25:10 UTC 2020
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 1, arcfour-hmac, no salt
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: [none]
Created 02-03-2020 09:49 AM
Hi @sergei_P ,
Could you please check if you have files in /etc/cloudera-scm-server folder named like cmf.principal and cmf.keytab? If so, please back them up in different folder and then delete them from /etc/cloudera-scm-server folder. After that, try your test again.
Thanks,
Li
Li Wang, Technical Solution Manager