Ranger KMS Login failure for keyadmin@AD.HADOOP.PRIVATE using password ********

[#1]

I have a kerberised cluster that uses AD. I have successfully installed Ranger and synced all users/groups specified. I am now working on using Ranger KMS but am running into some issues when I select the service in the Encryption tab.

When I select my cluster under "Select Service" the following error appears in a red pop up box:

Login failure for keyadmin@AD.HADOOP.PRIVATE using password ********

Here is the log output from xa_portal.log:

2016-05-23 12:02:08,290 [http-bio-6080-exec-2] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:64) - Request failed. SessionId=496, loginId=keyadmin, logMessage=Login failure for keyadmin@AD.HADOOP.PRIVATE using password ********
javax.ws.rs.WebApplicationException


2016-05-23 12:02:08,292 [http-bio-6080-exec-2] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:311) - Operation error. response=VXResponse={org.apache.ranger.view.VXResponse@279fbf8dstatusCode={1} msgDesc={Login failure for keyadmin@AD.HADOOP.PRIVATE using password ********} messageList={[VXMessage={org.apache.ranger.view.VXMessage@537d5281name={ERROR_SYSTEM} rbKey={xa.error.system} message={System Error. Please try later.} objectId={null} fieldName={null} }]} }
javax.ws.rs.WebApplicationException

Which configuration manages this credential?

I have a user created in AD called `keyadmin` however this user is not listed in the Ranger UI Users tab with all the other users... should it be?

What should the value of `hadoop.kms.authentication.kerberos.keytab` be? At the moment I have it set to /etc/security/keytabs/keyadmin.keytab.

Thanks.

[#2]

Hello @Dale Bradman

The correct values in the advance-kms site should be:

hadoop.kms.authentication.type=kerberos
hadoop.kms.authentication.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab
hadoop.kms.authentication.kerberos.principal=*

Also, the log messages in xa_portal.log doesn't look to be ERROR messages (they are INFO), so i believe that the root cause of error is somewhere above these lines. If they are not there, then I'd suggest you to enable debug logging for Ranger admin service and try the operation once again.

How to enable debug logging for Ranger admin service

STEP 1: On the Ranger admin host, edit the /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml file.

STEP 2: Change "info" to "debug" as shown in the below configuration stanza:
<category name="org.apache.ranger" additivity="false">
       <priority value="debug" />
       <appender-ref ref="xa_log_appender" />
</category>

STEP 3: Save and Restart the Ranger admin service.

[#3]

Thanks for your response @Vipin Rathor, I have changed to those configurations now.

But the same error appears in the Ranger UI and on the logs. Enabling debug logging doesn't actually produce anything else that is useful.

I have followed the Ranger KMS guide as much as possible. Could you confirm if the following configurations are correct please:

REPOSITORY_CONFIG_USERNAME = keyadmin@AD.HADOOP.PRIVATE
REPOSITORY_CONFIG_PASSWORD= password set in active directory for keyadmin user

Also, on the actual server that Ranger is hosted, if I ssh as keyadmin then the following appears:

root@dagobah:~# su keyadmin
kms@dagobah:~#

I have user sync set up with AD. The keyadmin username in AD is keyadmin not kms. Is this normal?

[#4 · Accepted solution]

@Vipin Rathor If these parameters are changed then Ambari will not update the KMS service in Ranger UI! You have to go into the UI and update the username/password fields to that of your AD yourself!!

REPOSITORY_CONFIG_USERNAME = keyadmin@AD.HADOOP.PRIVATE
REPOSITORY_CONFIG_PASSWORD= password set in active directory for keyadmin user

[#5]

Thanks @Dale Bradman for updating here. Glad that it worked for you.

[#6]

Hi @Dale Bradman,

I am also facing same issue, could you please let me know how did you resolve this issue?

Thanks

[#7]

See my answer above.

But basically, be careful with what you change in Ambari. Once Ambari is saved and Ranger KMS is restarted, it will not always update the same configurations that appear in the Ranger UI. So double check all your configs in Ranger UI.

Make sure your REPOSITORY_CONFIG_USERNAME=<principal>@AD.EXAMPLE in Ranger UI too

You're using kerberos and AD integration?

[#8]

Hi @Dale Bradman

Yes, Kerberos is enabled. It was working fine but after 2-3 days, it was start failing while connecting the repo and showing below message:-

2016-06-01 15:09:45,514 DEBUG RangerKmsAuthorizer - <== RangerkmsAuthorizer.hasAccess(GET_KEYS, keyadmin (auth:PROXY) via keyadmin@HDP-TBRND-DEV (auth:KERBEROS) , 😞 false 2016-06-01 15:10:02,625 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=tbarnd01_kms).loadPolicy()

If I will uninstall and reinstall it then it will work for 1-2 days then again start failing. It seems some principal is required kinit but not sure which one.

Do you have any suggestion? Please help me.

Thanks in advance.

[#9]

Interesting, not come across this before. Try checking the permissions of the keyadmin user in Ranger UI? Does it have get permissions?

Are you editing the Ranger KMS Ambari configs in this 1-2 day period?