Ranger policies failed to refresh after implementing Kerberos

[#1]

Hi guys,

Ranger fails to refresh policies after implementing Kerberos. I implemented Kerberos with new local MIT KDC, and using Ambari Automated Setup. HDFS, Hive and HBase works fine with new authentication method, but there are errors in refreshing policies. Every service where Ranger plugin is enabled gives me error:

2017-03-29 11:24:52,657 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(124)) - Error getting policies. secureMode=true, user=nn/hadoop1.locald@EXAMPLE.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=CLUSTER_hadoop
2017-03-29 11:24:52,657 ERROR util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(240)) - PolicyRefresher(serviceName=CLUSTER_hadoop): failed to refresh policies. Will continue to use last known version of policies (3)
java.lang.Exception: HTTP 401
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:217)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:185)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:158)

Thats for HDFS, for other services the user is different (hive etc.). I am using HDP 2.5 and Ambari 2.4.1.

These users exist in Kerberos (klist):

hive/hadoop1.locald@EXAMPLE.COM
hive/hadoop2.locald@EXAMPLE.COM
hive/hadoop3.locald@EXAMPLE.COM
hive/hadoop4.locald@EXAMPLE.COM
infra-solr/hadoop1.locald@EXAMPLE.COM
jhs/hadoop2.locald@EXAMPLE.COM
jn/hadoop1.locald@EXAMPLE.COM
jn/hadoop2.locald@EXAMPLE.COM
jn/hadoop3.locald@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/hadoop1.locald@EXAMPLE.COM
kafka/hadoop1.locald@EXAMPLE.COM
knox/hadoop1.locald@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
livy/hadoop1.locald@EXAMPLE.COM
livy/hadoop2.locald@EXAMPLE.COM
livy/hadoop4.locald@EXAMPLE.COM
nm/hadoop1.locald@EXAMPLE.COM
nm/hadoop2.locald@EXAMPLE.COM
nm/hadoop3.locald@EXAMPLE.COM
nm/hadoop4.locald@EXAMPLE.COM
nn/hadoop1.locald@EXAMPLE.COM
nn/hadoop2.locald@EXAMPLE.COM

[#2]

Ok, but what is the password for hdfs user?

I changed the user and password as it was shown here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/hdfs_plugin...

[#3]

you can give password of hdfs user , but that will not matter because in secure env keytab will be used , so just configure hdfs user and something in password it should work then

[#4]

Wanted to know which hdp version is this ?

[#5]

I Regenerated Keytabs once again and restarted all services and still not working, here my HDFS repo:

[#6]

can you also post value of hadoop.security.auth_to_local, because i see nn/hadoop1.locald@EXAMPLE.COM , it should be RULE:[2:$1@$0](nn@EXAMPLE.COM)s/.*/hdfs

[#7]

@Deepak Sharma it was only list of principles. My hadoop.security.auth_to_local is:

RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](amshbase@EXAMPLE.COM)s/.*/ams/
RULE:[2:$1@$0](amshbase@EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](amszk@EXAMPLE.COM)s/.*/ams/
RULE:[2:$1@$0](atlas@EXAMPLE.COM)s/.*/atlas/
RULE:[2:$1@$0](dn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](falcon@EXAMPLE.COM)s/.*/falcon/
RULE:[2:$1@$0](hbase@EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](hive@EXAMPLE.COM)s/.*/hive/
RULE:[2:$1@$0](jhs@EXAMPLE.COM)s/.*/mapred/
RULE:[2:$1@$0](jn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](knox@EXAMPLE.COM)s/.*/knox/
RULE:[2:$1@$0](livy@EXAMPLE.COM)s/.*/livy/
RULE:[2:$1@$0](nm@EXAMPLE.COM)s/.*/yarn/
RULE:[2:$1@$0](nn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](oozie@EXAMPLE.COM)s/.*/oozie/
RULE:[2:$1@$0](rangeradmin@EXAMPLE.COM)s/.*/ranger/
RULE:[2:$1@$0](rangerkms@EXAMPLE.COM)s/.*/keyadmin/
RULE:[2:$1@$0](rangerusersync@EXAMPLE.COM)s/.*/rangerusersync/
RULE:[2:$1@$0](rm@EXAMPLE.COM)s/.*/yarn/
RULE:[2:$1@$0](yarn@EXAMPLE.COM)s/.*/yarn/
DEFAULT

[#8]

Do you have HA enabled for Ranger?

[#9]

@vperiasamy I had but I deleted second Ranger Admin long time ago. Now I have single Ranger Admin server.

[#10]

@Edgar Daeds In case you are using HA, please make sure to add load balancer principal to spnego keytab. See steps 32 onwards in http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_hadoop-high-availability/content/configur...

Even with one Ranger admin, if you are seeing 401 in plugin side...

1] check if you have any error on ranger admin logs for authentication errors during policy download.

2] Verify keytab permissions.

3] Check policy.download.auth.users as mentioned by @Deepak Sharma above.

[#11]

@vperiasamy

I have HTTP/"host"@EXAMPLE.COM princs on all hosts.

Not even a single ERROR in xa_portal.log, only in services logs like HDFS or HIVE (posted above).

policy.download.auth.users is hdfs and hive for HDFS and Hive respectively (repo autocreated after disabling/enabling plugin).

Keytab permissions:

Also usersync is not syncing users:

30 Mar 2017 09:37:47 ERROR CustomPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User Group Info :
com.sun.jersey.api.client.UniformInterfaceException: POST http://myhost:6080/service/xusers/users/userinfo returned a response status of 401 Unauthorized
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568)
        at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.getUsergroupInfo(CustomPolicyMgrUserGroupBuilder.java:576)
        at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.access$500(CustomPolicyMgrUserGroupBuilder.java:77)
at
org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder$2.run(CustomPolicyMgrUserGroupBuilder.java:548)
at
org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder$2.run(CustomPolicyMgrUserGroupBuilder.java:544)
at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at
org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.addUserGroupInfo(CustomPolicyMgrUserGroupBuilder.java:544)
at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder.addOrUpdateUser(CustomPolicyMgrUserGroupBuilder.java:349) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomLdapUserGroupBuilder.updateSink(CustomLdapUserGroupBuilder.java:377) at org.apache.ranger.usergroupsync.UserGroupSync.syncUserGroup(UserGroupSync.java:114) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:87) at java.lang.Thread.run(Thread.java:745) 30 Mar 2017 09:37:47 INFO CustomLdapUserGroupBuilder [UnixUserSyncThread] - groupSearch is enabled, would search for groups and compute memberships 30 Mar 2017 09:37:47 INFO CustomLdapUserGroupBuilder [UnixUserSyncThread] - CustomLDAPUserGroupBuilder.getGroups() completed with group count: 0 30 Mar 2017 09:37:47 ERROR CustomPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.sun.jersey.api.client.UniformInterfaceException: POST http://192.168.3.82:6080/service/users/default returned a response status of 401 Unauthorized at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder.getMUser(CustomPolicyMgrUserGroupBuilder.java:847) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder.access$800(CustomPolicyMgrUserGroupBuilder.java:77) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder$5.run(CustomPolicyMgrUserGroupBuilder.java:820) at
org.apache.ranger.usergroupsync.UserGroupSync
.ldapsync.CustomPolicyMgrUserGroupBuilder$5.run(CustomPolicyMgrUserGroupBuilder.java:816) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.addMUser(CustomPolicyMgrUserGroupBuilder.java:816) at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.addOrUpdateUser(CustomPolicyMgrUserGroupBuilder.java:344) at org.apache.ranger.usergroupsync.UserGroupSync.CustomLdapUserGroupBuilder.updateSink(CustomLdapUserGroupBuilder.java:377) at org.apache.ranger.usergroupsync.UserGroupSync.syncUserGroup(UserGroupSync.java:114) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:87) at java.lang.Thread.run(Thread.java:745)