Error while enabling kerberos

[#1]

Hi,

I am trying to enable Kerberos for the newly built cluster, The user has full control over the OU and I also tested adding a user using ldapadd but cloudera manager throws INSUFF_ACCESS_RIGHTS for the same user:

2018-07-09 16:00:16,785 ERROR CommandPusher:com.cloudera.cmf.command.CommandHelpers: GenerateCredentials - Execution error: 
java.io.IOException: /usr/share/cmf/bin/gen_credentials_ad.sh failed with exit code 50 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf1473504603598890242.keytab
+ PRINC=hue/hostname.example.com@EXAMPLE.COM
+ USER=user_LFecFhBoWF
+ PASSWD=REDACTED
+ DELETE_ON_REGENERATE=true
+ SET_ENCRYPTION_TYPES=true
+ ENC_TYPES_MASK=28
+ USERACCOUNTCONTROL=66048
+ ACCOUNTEXPIRES=0
+ OBJECTCLASSES='objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
'
+ DIST_NAME='CN=prod_LFecFhBoWF,OU=username,OU=example Enterprise,DC=example,DC=com'
+ '[' -z /etc/krb5.conf ']'
+ echo 'Using custom config path '\''/etc/krb5.conf'\'', contents below:'
+ cat /etc/krb5.conf
+ SIMPLE_PWD_STR=
+ '[' 'REDACTED' = '' ']'
+ SIMPLE_PWD_STR='-x -D user@EXAMPLE>COM -w REDACTED'
++ mktemp /tmp/cm_ldap.XXXXXXXX
+ LDAP_CONF=/tmp/cm_ldap.9d2I7GO6
+ echo 'TLS_REQCERT never'
+ echo 'sasl_secprops minssf=0,maxssf=0'
+ export LDAPCONF=/tmp/cm_ldap.9d2I7GO6
+ LDAPCONF=/tmp/cm_ldap.9d2I7GO6
++ ldapsearch -LLL -H ldap://ldaphost.example.com -b 'OU=username,OU=Servers,OU=example Enterprise,DC=example,DC=com' -x -D user@EXAMPLE>COM -w 'REDACTED' userPrincipalName=hue/hostname.example.com@EXAMPLE.COM
+ PRINC_SEARCH=
++ echo ''
++ sed -n '1 {h; $ !d}; $ {x; s/\n //g; p}; /^ / {H; d}; /^ /! {x; s/\n //g; p}'
+ RESULTS_UNWRAPPED=
+ echo $'\342\200\234\342\200\235'
+ set +e
+ echo
+ grep -q userPrincipalName
+ '[' 1 -eq 0 ']'
+ set -e
+ '[' true = true ']'
+ ldapmodify -H ldap://ldaphost.example.com -x -D user@EXAMPLE>COM -w 'REDACTED'
++ echo 'objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
'
++ sed /str/d
++ echo hue/hostname.example.com@EXAMPLE.COM
++ sed -e 's/\@EXAMPLE.COM//g'
++ echo -n '"REDACTED"'
++ iconv -f UTF8 -t UTF16LE
++ base64 -w 0
ldap_add: Insufficient access (50)
additional info: 00000005: SecErr: DSID-031A11CC, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

>>

As soon as I hit Import KDC Account Manager Credentials, I also see the follwing message in /var/log/messages
Jul 9 17:17:39 hostname ldapsearch: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)

In need of serious help. Thanks in advance!

[#2]

@bgooley unicodePwd attribute in the /usr/share/cmf/bin/gen_credentials_ad.sh is causing the issue, 

Cloudera manager is still having the issues while creating the service principals, so I ran multiple tests in the same way as Cloudera manager by  changing/modifying attributes in LDIF file and figured out that unicodePwd property is throwing the below misleading error:

ldap_add: Insufficient access (50)
    additional info: 00000005: SecErr: DSID-031A11CC, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

I did a manual testing replacing the unicodePwd attribute with userPassword attribute in my test ldif file and it worked fine.....Can I make the change in /usr/share/cmf/bin/gen_credentials_ad.sh also..any idea on its consequences?

Thanks & Regards,

Siva

[#3]

@balusu,

It appears that you hav modified the script already to try to force it to use LDAP instead of LDAPS which I would guess is why the error 50 may be returned by Active Directory.

Either way, the main reason we did things this way is to protect the content of the password.  If you are shipping the password in the clear (LDAP) there is the opportunity that it could be intercepted and used to act on the cluster

It isn't "supported" but if you can get it to work that way, it's up to you.

If the change doesn't work, then you can always revert back, use LDAPS that we "hard coded" into the product and debug this further from the AD side.

let us know how it goes.

[#4]

@bgooley Thanks!! We are in the process of enabling ssl, Once done we will switch to ldaps.

I have changed the unicodepwd to userpasswd, Cloudera manager was able to create the principals, But while I was starting the services, zookeeper threw the following error:

5:05:02.212 PM

ERROR

QuorumPeerMain

Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: ICMP Port Unreachable at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

[#5]

@balusu,

That's great news and the new issue is something completely unrelated it seems.

When zookeeper tries to login "kinit" that fails since it cannot reach the port.

By default Kerberos uses UDP, so I wonder if UDP packets are being blocked when going to your KDC.

I would try the following:

klist -kt /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep zookeeper| tail -1`/zookeeper.keytab

This should show you the zookeer principal

Try running "kinit" like this:

Note the principal name.

kinit -kt /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep zookeeper| tail -1`/zookeeper.keytab principal_from_klist_output

Even if this succeeds, I'd consider forcing Kerberos clients to use TCP just to see by adding this in the [libdefaults] section of your /etc/krb5.conf on the zookeeper host:

udp_preference_limit=1

Hope some of this helps you track down the cause.

[#6]

@bgooley 

kinit -kt /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep zookeeper| tail -1`/zookeeper.keytab zookeeper/host@EXAMPLE.COM

kinit: Client's credentials have been revoked while getting initial credentials

I was able to do a kinit as my self from the same host and There are no connection issues. Never spent this much time on enabling kerberos 😛

[#7]

@balusu,

Good testing and solid results.

So this means that Active Directory has disabled the account.  You may need to check the Event Viewer's Security entries to find out why.  You may check the zookeeper credential object in AD and verify that it is enabled, never expires, password doesn't need to be changed, etc.

[#8]

@bgooley

I was able to create the principles when I commented out UserAccountControl, But the problem with this is the account getting locked and the below property set:

If use UserAccountControl attribute I get

ldap_add: Server is unwilling to perform (53)

additional info: 0000052D: SvcErr: DSID-031A1248, problem 5003 (WILL_NOT_PERFORM), data 0

But when creating the user through AD UI everything is good, That states that there is no access/policy blocking issue. It seems conflict between openldap vs AD. Any suggestions/directions for more testing?

Thanks,

Siva

[#9]

@balusu,

The script works without manipulation but it requires LDAPS, so either you have to figure outall these nuances of how to get it to work without LDAPS or you could look at alternatives:

(1)

Create all the principal objects yourself and implement a keytab retrieval script as described here:

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/sg_keytab_retrieval_script.html

(2)

Install and configure an MIK Kerberos KDC until you can get LDAPS support in AD.  Changing from MIT KDC to AD KDC is not too hard, but does require some downtime.

[#10]

@bgooley 

ldaps:// smoothly generated the accounts,  unicodepwd, usersharedcontrol must be needed for AD to create users and for those ldaps is mandatory. I think it should be added to Cloudera documentation.

kinit -kt /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep zookeeper| tail -1`/zookeeper.keytab principal

[root@hostname sbalusu_c@example.com]# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: zookeeper/hostname.example.com@EXAMPLE.COM

Valid starting Expires Service principal
07/12/2018 16:12:00 07/13/2018 02:12:00 krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 07/19/2018 16:11:59

I have added udp_preference_limit=1 but still getting the earlier error:

4:17:47.210 PM

ERROR

QuorumPeerMain

Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: ICMP Port Unreachable at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

Documentation does not talk about ICMP port, I tried enabling debug mode but could not figure out which port it is looking for. Any idea?

Thanks & Regards,

Siva