Ranger Admin policies not synced for all services

[#1]

Hi, I have a problem with policy synchronization. HDP 2.5.0-1245, Ambari 2.4.1. When I add/edit policy, the policy changes in MySQL database "ranger", but not in Audit -> Plugins. Also the change does not affect on /etc/ranger/servicename/policycache/XXX.json file (last change was 10 days ago, same as Plugin tab shows). From xa_portal.log in DEBUG mode I see an error every 30 seconds:

"DEBUG org.springframework.security.web.access.ExceptionTranslationFilter (ExceptionTranslationFilter.java:165) - Access is denied (user is anonymous); redirecting to authentication entry point org springframework.security.access.AccessDeniedException: Access is denied"

That is the only difference in this log when I compare my working cluster with this one.

The connection to database is good, but what happens next I dont know. How do policies are propagated from MySQL to .json file? Which user is responsible for this? Any solution ideas? Thank you in advance.

[#2 · Accepted solution]

ranger plugin should be enabled for corresponding service for downloading the policies, and service(eg. hdfs,hbase) keep on pulling the policies from ranger every 30 sec. so i think if you have enabled then plugin then there should be some issue while pulling the policies by service from the ranger.

can your please check the corresponding service logs for the issue, and even in ranger access audit logs keep on publishing the information about this call there also you check if response for policy download is 200/304.

[#3]

@Deepak Sharma

Thank you for a quick answer.

In access_log when I edit a policy I got PUT and GET code 200, but except that every 30 seconds I got GET with code 302.

In correspoding service logs I got "failed to refresh policies. Will continue to use last known version of policies (61).

It was working before, but for 10 days policies are not synced.

[#4]

can you check if ranger policy url is properly defined in hdfs ranger plugin config, and was there anything changed when you started seeing this issue ?

[#5]

Sorry guys, that was the issue. Not only me have access to that cluster, and somebody changed ranger policy url in every plugin config. Thank you!

[#6]

The strange thing is that in access_log Ranger tries to GET status from not defined URL:

"IP - - [DATE] "GET / HTTP/1.0" 302 -"

[#7]

@Edgar Daeds

Can you check below things -

1. is your test connection for repository work successfully ?

2. Do you see any alert in Ambari UI wrt "Ranger Admin Password check"?

3. If that is HDFS policy can you check namenode logs if you any error there ?

Please post more logs to analyze the issue.

[#8]

@Sagar Shimpi

1. Test connection works for HDFS and HBase, for Hive I have my own modified jars and it is not working for a long time (but policies were working)

2. In Ambari UI I dont see any alerts regarding to Ranger

3. In every service logs I see only one error which is known: "failed to refresh policies. Will continue to use last known version of policies (XX) [...] Connection Refused".

Could you please specify which logs I can post? It is on clients cluster and I dont have direct access, so I need to rewrite logs manually.

P. S. policies are not synced for every service which I am using (HDFS,HBase,Hive,Kafka,Knox)

[#9]

If this is kerberos env, make sure there are no kerberos ticket renewal errors.

Also, are you using SSL for ranger? In that case, make sure plugins are configured to trust ranger admin certificate.