question Re: Ranger policies failed to refresh after implementing Kerberos in Support Questions

Ranger policies failed to refresh after implementing Kerberos

frank93 — Wed, 29 Mar 2017 17:03:22 GMT

Hi guys,

Ranger fails to refresh policies after implementing Kerberos. I implemented Kerberos with new local MIT KDC, and using Ambari Automated Setup. HDFS, Hive and HBase works fine with new authentication method, but there are errors in refreshing policies. Every service where Ranger plugin is enabled gives me error:

2017-03-29 11:24:52,657 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(124)) - Error getting policies. secureMode=true, user=nn/hadoop1.locald@EXAMPLE.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=CLUSTER_hadoop
2017-03-29 11:24:52,657 ERROR util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(240)) - PolicyRefresher(serviceName=CLUSTER_hadoop): failed to refresh policies. Will continue to use last known version of policies (3)
java.lang.Exception: HTTP 401
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:217)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:185)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:158)

Thats for HDFS, for other services the user is different (hive etc.). I am using HDP 2.5 and Ambari 2.4.1.

These users exist in Kerberos (klist):

hive/hadoop1.locald@EXAMPLE.COM
hive/hadoop2.locald@EXAMPLE.COM
hive/hadoop3.locald@EXAMPLE.COM
hive/hadoop4.locald@EXAMPLE.COM
infra-solr/hadoop1.locald@EXAMPLE.COM
jhs/hadoop2.locald@EXAMPLE.COM
jn/hadoop1.locald@EXAMPLE.COM
jn/hadoop2.locald@EXAMPLE.COM
jn/hadoop3.locald@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/hadoop1.locald@EXAMPLE.COM
kafka/hadoop1.locald@EXAMPLE.COM
knox/hadoop1.locald@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
livy/hadoop1.locald@EXAMPLE.COM
livy/hadoop2.locald@EXAMPLE.COM
livy/hadoop4.locald@EXAMPLE.COM
nm/hadoop1.locald@EXAMPLE.COM
nm/hadoop2.locald@EXAMPLE.COM
nm/hadoop3.locald@EXAMPLE.COM
nm/hadoop4.locald@EXAMPLE.COM
nn/hadoop1.locald@EXAMPLE.COM
nn/hadoop2.locald@EXAMPLE.COM

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Sun, 18 Aug 2019 09:13:47 GMT

can you please check if ranger is also kerberised , because if it is hdp2.5 or above then it will be kerberised.

if it is then can you please try following

1) regenerating keytabs from ambari and restart the services.

2) add following properties in the repos on ranger:

policy.grantrevoke.auth.users: hbase ( or corresponding service user)

tag.download.auth.users: hbase ( or corresponding service user)

policy.download.auth.users: hbase (or corresponding service user)

same way these properties to be added in hdfs repo too , and service user will be hdfs or what ever you have in your cluster.

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Wed, 29 Mar 2017 17:47:35 GMT

@Deepak Sharma thank you for a quick answer. Ranger is also Kerberized. I added those properties and changed Authentication Type in HDFS Repo to Kerberos. Now Test connection is done successfully, but the same error appears. After these changes few INFO logs appeared:

2017-03-29 12:46:23,368 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(124)) - Error getting policies. secureMode=true, user=nn/hadoop1.locald@EXAMPLE.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=3SOFT_HDL_hadoop
2017-03-29 12:46:23,368 ERROR util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(240)) - PolicyRefresher(serviceName=3SOFT_HDL_hadoop): failed to refresh policies. Will continue to use last known version of policies (3)
java.lang.Exception: HTTP 401
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:217)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:185)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:158)
2017-03-29 12:46:24,577 WARN  protocol.ResponseProcessCookies (ResponseProcessCookies.java:processCookies(122)) - Cookie rejected [hadoop.auth="", version:0, domain:hadoop1.locald, path:/, expiry:Thu Jan 01 01:00:00 CET 1970] Domain attribute "hadoop1.locald" violates the Netscape cookie specification
2017-03-29 12:46:24,582 WARN  protocol.ResponseProcessCookies (ResponseProcessCookies.java:processCookies(122)) - Cookie rejected [hadoop.auth=""u=nn&p=nn/hadoop1.locald@EXAMPLE.COM&t=kerberos&e=1490820384581&s=hi0THf8d5c4wUgzQbs/+W/PENPo="", version:0, domain:hadoop1.locald, path:/, expiry:Wed Mar 29 22:46:24 CEST 2017] Domain attribute "hadoop1.locald" violates the Netscape cookie specification
2017-03-29 12:46:25,229 INFO  BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1580)) - BLOCK* neededReplications = 0, pendingReplications = 0.
2017-03-29 12:46:27,578 WARN  protocol.ResponseProcessCookies (ResponseProcessCookies.java:processCookies(122)) - Cookie rejected [hadoop.auth="", version:0, domain:hadoop1.locald, path:/, expiry:Thu Jan 01 01:00:00 CET 1970] Domain attribute "hadoop1.locald" violates the Netscape cookie specification
2017-03-29 12:46:27,582 WARN  protocol.ResponseProcessCookies (ResponseProcessCookies.java:processCookies(122)) - Cookie rejected [hadoop.auth=""u=nn&p=nn/hadoop1.locald@EXAMPLE.COM&t=kerberos&e=1490820387581&s=S0zta5LH3SfBXFh0XoB3T5ldjsQ="", version:0, domain:hadoop1.locald, path:/, expiry:Wed Mar 29 22:46:27 CEST 2017] Domain attribute "hadoop1.locald" violates the Netscape cookie specification
2017-03-29 12:46:28,230 INFO  BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1580)) - BLOCK* neededReplications = 0, pendingReplications = 0.
2017-03-29 12:46:28,474 INFO  ipc.Server (Server.java:saslProcess(1538)) - Auth successful for nn/hadoop1.locald@EXAMPLE.COM (auth:KERBEROS)
2017-03-29 12:46:28,475 INFO  authorize.ServiceAuthorizationManager (ServiceAuthorizationManager.java:authorize(137)) - Authorization successful for nn/hadoop1.locald@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Sun, 18 Aug 2019 09:13:39 GMT

did you regenerated keytabs and restarted service?

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Wed, 29 Mar 2017 18:20:33 GMT

Yes, I regenerated keytabs and restarted services. I dont get it: The log:

2017-03-29 13:26:35,429 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(124)) - Error getting policies. secureMode=true, user=nn/hadoop1.locald@EXAMPLE.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=CLUSTER_hadoop
2017-03-29 13:26:35,429 ERROR util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(240)) - PolicyRefresher(serviceName=CLUSTER_hadoop): failed to refresh policies. Will continue to use last known version of policies (3)
java.lang.Exception: HTTP 401

says user nn/hadoop1.locald@EXAMPLE.COM us unauthorized (HTTP 401), but below is:

2017-03-29 13:26:38,877 INFO  ipc.Server (Server.java:saslProcess(1538)) - Auth successful for nn/hadoop1.locald@EXAMPLE.COM (auth:KERBEROS)

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Wed, 29 Mar 2017 18:49:56 GMT

can you check hadoop.security.auth_to_local config in hdfs & hdfs repo also , if rule is specified for nn ,

RULE:[2:$1@$0](nn@EXAMPLE.COM)s/.*/hdfs/

so that call is sent as hdfs user , and since hdfs user is in policy,download.auth.users so it will be alllowed to download the policy and make sure same config is pres in hdfs repo config also

check this config:

RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](activity_analyzer@EXAMPLE.COM)s/.*/activity_analyzer/
RULE:[2:$1@$0](activity_explorer@EXAMPLE.COM)s/.*/activity_explorer/
RULE:[2:$1@$0](amshbase@EXAMPLE.COM)s/.*/ams/
RULE:[2:$1@$0](amszk@EXAMPLE.COM)s/.*/ams/
RULE:[2:$1@$0](atlas@EXAMPLE.COM)s/.*/atlas/
RULE:[2:$1@$0](dn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](hbase@EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](hive@EXAMPLE.COM)s/.*/hive/
RULE:[2:$1@$0](jhs@EXAMPLE.COM)s/.*/mapred/
RULE:[2:$1@$0](knox@EXAMPLE.COM)s/.*/knox/
RULE:[2:$1@$0](nfs@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](nm@EXAMPLE.COM)s/.*/yarn/
RULE:[2:$1@$0](nn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](rangeradmin@EXAMPLE.COM)s/.*/ranger/
RULE:[2:$1@$0](rangertagsync@EXAMPLE.COM)s/.*/rangertagsync/
RULE:[2:$1@$0](rangerusersync@EXAMPLE.COM)s/.*/rangerusersync/
RULE:[2:$1@$0](rm@EXAMPLE.COM)s/.*/yarn/
RULE:[2:$1@$0](yarn@EXAMPLE.COM)s/.*/yarn/
DEFAULT

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Wed, 29 Mar 2017 18:57:27 GMT

I have exactly the same rules that you uploaded, both in hdfs and hdfs repo. I deleted my old repo and let Ambari create new one, and the newly created HDFS repo has correct configs and test connection is done successfully.

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Wed, 29 Mar 2017 19:01:41 GMT

can you share screenshot of your repo config , I want to see which is the repo user ?

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Sun, 18 Aug 2019 09:13:31 GMT

and can you please change the repo user to hdfs if it is something else

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Wed, 29 Mar 2017 19:13:42 GMT

Ok, but what is the password for hdfs user?

I changed the user and password as it was shown here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/hdfs_plugin_kerberos.html

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Sun, 18 Aug 2019 09:13:18 GMT

@Deepak Sharma Sorry I missed this comment, here is my config:

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Wed, 29 Mar 2017 19:43:49 GMT

you can give password of hdfs user , but that will not matter because in secure env keytab will be used , so just configure hdfs user and something in password it should work then

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Sun, 18 Aug 2019 09:13:10 GMT

I Regenerated Keytabs once again and restarted all services and still not working, here my HDFS repo:

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Wed, 29 Mar 2017 21:11:08 GMT

Wanted to know which hdp version is this ?

Re: Ranger policies failed to refresh after implementing Kerberos

dsharma — Wed, 29 Mar 2017 21:15:53 GMT

can you also post value of hadoop.security.auth_to_local, because i see nn/hadoop1.locald@EXAMPLE.COM , it should be RULE:[2:$1@$0](nn@EXAMPLE.COM)s/.*/hdfs

Re: Ranger policies failed to refresh after implementing Kerberos

vperiasamy — Wed, 29 Mar 2017 21:22:59 GMT

Do you have HA enabled for Ranger?

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Wed, 29 Mar 2017 21:56:21 GMT

@Deepak Sharma it was only list of principles. My hadoop.security.auth_to_local is:

RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](amshbase@EXAMPLE.COM)s/.*/ams/
RULE:[2:$1@$0](amshbase@EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](amszk@EXAMPLE.COM)s/.*/ams/
RULE:[2:$1@$0](atlas@EXAMPLE.COM)s/.*/atlas/
RULE:[2:$1@$0](dn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](falcon@EXAMPLE.COM)s/.*/falcon/
RULE:[2:$1@$0](hbase@EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](hive@EXAMPLE.COM)s/.*/hive/
RULE:[2:$1@$0](jhs@EXAMPLE.COM)s/.*/mapred/
RULE:[2:$1@$0](jn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](knox@EXAMPLE.COM)s/.*/knox/
RULE:[2:$1@$0](livy@EXAMPLE.COM)s/.*/livy/
RULE:[2:$1@$0](nm@EXAMPLE.COM)s/.*/yarn/
RULE:[2:$1@$0](nn@EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](oozie@EXAMPLE.COM)s/.*/oozie/
RULE:[2:$1@$0](rangeradmin@EXAMPLE.COM)s/.*/ranger/
RULE:[2:$1@$0](rangerkms@EXAMPLE.COM)s/.*/keyadmin/
RULE:[2:$1@$0](rangerusersync@EXAMPLE.COM)s/.*/rangerusersync/
RULE:[2:$1@$0](rm@EXAMPLE.COM)s/.*/yarn/
RULE:[2:$1@$0](yarn@EXAMPLE.COM)s/.*/yarn/
DEFAULT

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Wed, 29 Mar 2017 21:57:41 GMT

@vperiasamy I had but I deleted second Ranger Admin long time ago. Now I have single Ranger Admin server.

Re: Ranger policies failed to refresh after implementing Kerberos

vperiasamy — Wed, 29 Mar 2017 22:53:28 GMT

@Edgar Daeds In case you are using HA, please make sure to add load balancer principal to spnego keytab. See steps 32 onwards in http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_hadoop-high-availability/content/configure_ranger_admin_ha.html#configure_ranger_admin_ha_without_ssl

Even with one Ranger admin, if you are seeing 401 in plugin side...

1] check if you have any error on ranger admin logs for authentication errors during policy download.

2] Verify keytab permissions.

3] Check policy.download.auth.users as mentioned by @Deepak Sharma above.

Re: Ranger policies failed to refresh after implementing Kerberos

frank93 — Sun, 18 Aug 2019 09:13:02 GMT

@vperiasamy

I have HTTP/"host"@EXAMPLE.COM princs on all hosts.

Not even a single ERROR in xa_portal.log, only in services logs like HDFS or HIVE (posted above).

policy.download.auth.users is hdfs and hive for HDFS and Hive respectively (repo autocreated after disabling/enabling plugin).

Keytab permissions:

Also usersync is not syncing users:

30 Mar 2017 09:37:47 ERROR CustomPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User Group Info :
com.sun.jersey.api.client.UniformInterfaceException: POST http://myhost:6080/service/xusers/users/userinfo returned a response status of 401 Unauthorized
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568)
        at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.getUsergroupInfo(CustomPolicyMgrUserGroupBuilder.java:576)
        at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.access$500(CustomPolicyMgrUserGroupBuilder.java:77)
at
org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder$2.run(CustomPolicyMgrUserGroupBuilder.java:548)
at
org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder$2.run(CustomPolicyMgrUserGroupBuilder.java:544)
at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at
org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.addUserGroupInfo(CustomPolicyMgrUserGroupBuilder.java:544)
at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder.addOrUpdateUser(CustomPolicyMgrUserGroupBuilder.java:349) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomLdapUserGroupBuilder.updateSink(CustomLdapUserGroupBuilder.java:377) at org.apache.ranger.usergroupsync.UserGroupSync.syncUserGroup(UserGroupSync.java:114) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:87) at java.lang.Thread.run(Thread.java:745) 30 Mar 2017 09:37:47 INFO CustomLdapUserGroupBuilder [UnixUserSyncThread] - groupSearch is enabled, would search for groups and compute memberships 30 Mar 2017 09:37:47 INFO CustomLdapUserGroupBuilder [UnixUserSyncThread] - CustomLDAPUserGroupBuilder.getGroups() completed with group count: 0 30 Mar 2017 09:37:47 ERROR CustomPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.sun.jersey.api.client.UniformInterfaceException: POST http://192.168.3.82:6080/service/users/default returned a response status of 401 Unauthorized at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder.getMUser(CustomPolicyMgrUserGroupBuilder.java:847) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder.access$800(CustomPolicyMgrUserGroupBuilder.java:77) at
org.apache.ranger.usergroupsync.UserGroupSync
.CustomPolicyMgrUserGroupBuilder$5.run(CustomPolicyMgrUserGroupBuilder.java:820) at
org.apache.ranger.usergroupsync.UserGroupSync
.ldapsync.CustomPolicyMgrUserGroupBuilder$5.run(CustomPolicyMgrUserGroupBuilder.java:816) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.addMUser(CustomPolicyMgrUserGroupBuilder.java:816) at org.apache.ranger.usergroupsync.UserGroupSync.CustomPolicyMgrUserGroupBuilder.addOrUpdateUser(CustomPolicyMgrUserGroupBuilder.java:344) at org.apache.ranger.usergroupsync.UserGroupSync.CustomLdapUserGroupBuilder.updateSink(CustomLdapUserGroupBuilder.java:377) at org.apache.ranger.usergroupsync.UserGroupSync.syncUserGroup(UserGroupSync.java:114) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:87) at java.lang.Thread.run(Thread.java:745)