Reply
Contributor
Posts: 33
Registered: ‎04-26-2017

Sentry HDFS ACL Sync with Hive HA Problems

[ Edited ]

Since going to 5.13 I enabled Sentry HDFS ACL sync and also sentry HA, now that it's compatable with hive metastore ha.  Since doing this though i'm seeing lots of errors in the hive log, as well as canary failing to create databases and or partitions at random times.

Due to the metastore having problems, this impacts impala etc.

The hive error logs shows the following errors:

2017-12-15 14:39:13,093 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-5-thread-46]: Failed to sync requested HMS notifications up to the event ID: 819694
2017-12-15 14:39:13,103 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: failed to execute syncNotifications
2017-12-15 14:39:13,104 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: Thrift call failed
2017-12-15 14:40:53,180 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: failed to execute syncNotifications
2017-12-15 14:40:53,181 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: Thrift call failed
2017-12-15 14:40:53,220 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: failed to execute syncNotifications
2017-12-15 14:40:53,221 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: Thrift call failed
2017-12-15 14:42:33,111 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: failed to execute syncNotifications
2017-12-15 14:42:33,112 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: Thrift call failed
2017-12-15 14:44:13,189 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: failed to execute syncNotifications
2017-12-15 14:44:13,189 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: Thrift call failed
2017-12-15 14:44:13,225 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: failed to execute syncNotifications
2017-12-15 14:44:13,225 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: Thrift call failed
2017-12-15 14:44:13,226 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-5-thread-47]: Failed to sync requested HMS notifications up to the event ID: 819785

And the sentry service is full of the following:

2017-12-15 14:45:53,199 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.
2017-12-15 14:46:18,056 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.
2017-12-15 14:47:33,093 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.


2017-12-15 14:56:41,602 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:44,617 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:47,630 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:50,641 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:53,654 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!


Any ideas anyone?

Highlighted
New Contributor
Posts: 1
Registered: ‎10-24-2014

Re: Sentry HDFS ACL Sync with Hive HA Problems

I ran into a similar issue that was also logging this:

ERROR org.apache.sentry.service.thrift.HMSFollower: Failed to get the last processed notification id from sentry store, Skipping the processing
javax.jdo.JDODataStoreException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1

If you're seeing something similar to that, check your mysql-connector-java.jar version on all of your hosts, it may need updating.

New Contributor
Posts: 3
Registered: ‎05-11-2016

Re: Sentry HDFS ACL Sync with Hive HA Problems

Did you solve this issue? I am facing the same issue.

Contributor
Posts: 33
Registered: ‎04-26-2017

Re: Sentry HDFS ACL Sync with Hive HA Problems

No, still outstanding.  Will try enabling again after upgrading to 5.14

Announcements