Created on 11-29-2016 09:12 AM - edited 08-19-2019 02:59 AM
Hello
We use Ranger for auditing Hive. Yesterday at 15:20 after restarting the Server that runs Ambari (Ambari audits to the default mysql which is on the same server) the auditing has stopped working. Ranger is up & running, mysql is also running, plugins are working and synced with Hiveserver2. Yet no data is being audited to he db since the reboot of the server. I've queried the xa_ranger_audit table and it shows entries up until the server reboot.
Clearly something imperative is not running.
Any suggestions are appreciated !
Created 11-29-2016 05:40 PM
Found the problem.
Investigating the hiveserver2.log showed that rangerlogger failed to flush data to the db due to permission problem (Internal Exception: java.sql.SQLException: Access denied for user 'rangerlogger'@'<server>' (using password: YES) Error Code: 1045)
After granting the permissions it flushed the data successfully.
Now i just need to figure out what caused the loss of permissions and how the hell it is related to the server reboot....
Thanks for all your help !
Created 11-29-2016 09:19 AM
Can you check if you see any error in xa_portal.log?
Can you try disabling the plugin from configs for respective services and re-enable?
Please try to enable ranger debug -
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Menlo} span.s1 {font-variant-ligatures: no-common-ligatures}
/usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml
Change From: <category name="org.apache.ranger" additivity="false"> <priority value="info" /> <appender-ref ref="xa_log_appender" /> </category> Change To: <category name="org.apache.ranger" additivity="false"> <priority value="debug" /> <appender-ref ref="xa_log_appender" /> </category>
Restart Ranger admin and check the xa_portal logs again.
Created 11-29-2016 12:13 PM
@Sagar Shimpi Thank you for replying. I wanted to add that login sessions are being audited. It's just the Access information (Hive access) ism't audited. I enabled debug as per your recommendation and restarted ranger but haven't found any errors regarding audit or connection to db. There is an LDAP error: DEBUG org.apache.ranger.security.handler.RangerAuthenticationProvider (RangerAuthenticationProvider.java:280) - AD Authentication Failed: org.springframework.security.authentication.BadCredentialsException: Bad credentials
But it doesn't say which user and i'm not sure it is related. Our Hive is configured for LDAP auth so maybe it is related.
Created 11-29-2016 09:23 AM
please check properties:
- xasecure.audit.destination.db.user
- xasecure.audit.destination.db.password
- xasecure.audit.destination.db.jdbc.url
- xasecure.audit.destination.db.jdbc.driver
and whether "Audit to DB" is checked in Ambari -> Hive -> Configs -> Advanced -> Advanced ranger-hive-audit. Try to type your own values instead of {{variable}}.
Created 11-29-2016 12:15 PM
Thank you for replying.
Audit to DB is checked in Ambari. I also unchecked, saved and restarted, then checked again, saved and restarted - but still problem remains.
Regarding using own values instead of variables - i'm not sure exactly what to enter and again - until the reboot it worked flawlessly.
Created 11-29-2016 12:20 PM
I had a similar problem to yours once (works till reboot). I typed the values of these parameters:
- xasecure.audit.destination.db.user
- xasecure.audit.destination.db.password
as "plain text" and it worked for me. First one is a MySQL Username which has access to ranger_audit database and second parameter is his password.
Created 11-29-2016 03:58 PM
I've changed those settings to clear text and restarted hive + Ranger but problem remains unfortunately.
Created 11-29-2016 12:23 PM
One more thing to add: we restarted the server after installing and configuring sssd on the server for group mapping.
Created 11-29-2016 01:46 PM
If ranger is integrated with ldap then you need to revisit the config.
Created 11-29-2016 03:43 PM
Can you be more specific ? The Ranger authentication is LDAP, meaning in order to login to it's UI you can only login using your Active Directory credentials. This works without any problems.