question Re: Kerberos wizard does not Kerberize Ambari in Support Questions

Kerberos wizard does not Kerberize Ambari

qiwang — Fri, 23 Dec 2016 02:46:55 GMT

Ambari 2.4.2.0 and HDP 2.5.3.0-37

Running Kerberos wizard to Kerberize the cluster, all HDP components are Kerberized successfully, but Ambari itself is not Kerberized. I can see the ambari-server principle being added to MIT KDC, but there is no keytab for that principle under /etc/security/keytab on Ambari server. This cause all the views not working.

I did the manual step to Kerberize Ambari and everything is fine afterward. I am pretty sure back in the last version of Ambari, it was Kerberized by the wizard. Not sure if this is a change of behavior or some other things caused it. Ambari server was running under ambari-server rather than root account before Kerberos wizard, not sure if that change anything.

Re: Kerberos wizard does not Kerberize Ambari

rlevas — Fri, 23 Dec 2016 05:01:12 GMT

@Qi Wang

Since Ambari is running a non-privileged user, it is possible that the keytab file failed to be moved into place due to permission issues. Make sure that the sudoers file is setup properly - for example, see http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-security/content/sudoer_configuration_server.html

This is needed due to a change made for Ambari 2.4.0 where Ambari, by default, creates it own Kerberos identity and distribute its keytab file to itself. Sudo is needed to this due to permission issues writing to /etc/security/keytabs.

To confirm this is the issue, you might want to take a look at the ambari-server.log to see if there are any relevant error posted. I, unfortunately, do know the exact wording to look for off the top of my head.

Re: Kerberos wizard does not Kerberize Ambari

dvillarreal — Fri, 23 Dec 2016 05:01:55 GMT

In 2.4.2 you have to manually setup Ambari principal and keytab

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/_set_up_kerberos_for_ambari_server.html

I see the same documentation for 2.5.3.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_set_up_kerberos_for_ambari_server.html

Re: Kerberos wizard does not Kerberize Ambari

qiwang — Fri, 23 Dec 2016 05:10:21 GMT

look at the note on 2.5.3 doc, "If you performed the Automated Kerberos Setup, these steps are performed automatically (and therefore, you do not need to perform the steps below)."

Re: Kerberos wizard does not Kerberize Ambari

dvillarreal — Fri, 23 Dec 2016 05:49:24 GMT

However, in Ambari 2.4.x and up it should create the principal and keytabs automatically. I have seen where this didn't happen prior to 2.4.2 on 2.4.0.1 and 2.4.1

Re: Kerberos wizard does not Kerberize Ambari

qiwang — Fri, 23 Dec 2016 05:56:45 GMT

If the non-root user missing permission, it still get all the keytab for other users copied and chown done. Not sure why it only failed on Ambari itself. Will try do this again and look at the log to see if there is any abnormally.

Re: Kerberos wizard does not Kerberize Ambari

rlevas — Fri, 23 Dec 2016 06:04:57 GMT

The Ambari agent handles the other keytab files. If this is working, then it is either running at root or sudo has been properly set up for it.

Re: Kerberos wizard does not Kerberize Ambari

qiwang — Fri, 23 Dec 2016 10:35:39 GMT

I reproduced the same problem again.

1. change ambari to run with non-root before Kerberos, (also change it to run HTTPS, encrypt password for ambari and sync ldap user)

2. Kerberos wizard with MIT KDC

In log I found

23 Dec 2016 01:45:51,007  INFO [Server Action Executor Worker 333] CreateKeytabFilesServerAction:193 - Creating keytab file for ambari-server@FIELD.HORTONWORKS.COM on host ambari_server

So look like the process did try to create the keytab

But under /etc/security/keytab, there is no ambari.server.keytab, I also try find any warning or error in ambari-server.log that indicates anything went wrong, but see nothing related.

And also include the cmd for creating the non-root user for ambari-server, ambari-agent is still running under root user

useradd -d /var/lib/ambari-server -G hadoop -M -r -s /sbin/nologin ambari-user
echo 'ambari-user ALL=(ALL) NOPASSWD:SETENV: /bin/mkdir, /bin/cp, /bin/chmod, /bin/rm' > /etc/sudoers.d/ambari-server

Re: Kerberos wizard does not Kerberize Ambari

rlevas — Fri, 23 Dec 2016 22:58:08 GMT

After the "Create Keytabs" stage, is there a "Configure Ambari Identity" stage? If so, did it pass? If not, maybe your sudo rules are incorrect. Trying yours

ambari-user ALL=(ALL) NOPASSWD:SETENV: /bin/mkdir, /bin/cp, /bin/chmod, /bin/rm

My "Configure Ambari Identity" stage failed.

Changing the rule to

ambari-user ALL=(ALL) NOPASSWD:SETENV: /bin/mkdir, /bin/cp, /bin/chmod, /bin/chown, /bin/chgroup, /bin/rm

did the trick for me.

Re: Kerberos wizard does not Kerberize Ambari

qiwang — Sat, 24 Dec 2016 00:51:53 GMT

There was a failure during "Configure Ambari Identity", but retry passed. So I thought it was not really a problem. I am sure the sudo rule is the problem. Will try again and let you know the outcome.

Update the sudo permission and got another error: "you must have a tty to run sudo". this turns out to be related to sudo setting, use visudo to comment requiretty fixed the problem

visudo
#Defaults    requiretty