Since going to 5.13 I enabled Sentry HDFS ACL sync and also sentry HA, now that it's compatable with hive metastore ha.  Since doing this though i'm seeing lots of errors in the hive log, as well as canary failing to create databases and or partitions at random times.

Due to the metastore having problems, this impacts impala etc.

The hive error logs shows the following errors:

2017-12-15 14:39:13,093 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-5-thread-46]: Failed to sync requested HMS notifications up to the event ID: 819694
2017-12-15 14:39:13,103 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: failed to execute syncNotifications
2017-12-15 14:39:13,104 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: Thrift call failed
2017-12-15 14:40:53,180 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: failed to execute syncNotifications
2017-12-15 14:40:53,181 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: Thrift call failed
2017-12-15 14:40:53,220 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: failed to execute syncNotifications
2017-12-15 14:40:53,221 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: Thrift call failed
2017-12-15 14:42:33,111 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: failed to execute syncNotifications
2017-12-15 14:42:33,112 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: Thrift call failed
2017-12-15 14:44:13,189 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: failed to execute syncNotifications
2017-12-15 14:44:13,189 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: Thrift call failed
2017-12-15 14:44:13,225 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: failed to execute syncNotifications
2017-12-15 14:44:13,225 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: Thrift call failed
2017-12-15 14:44:13,226 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-5-thread-47]: Failed to sync requested HMS notifications up to the event ID: 819785

And the sentry service is full of the following:

2017-12-15 14:45:53,199 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.
2017-12-15 14:46:18,056 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.
2017-12-15 14:47:33,093 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.


2017-12-15 14:56:41,602 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:44,617 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:47,630 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:50,641 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:53,654 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!

Any ideas anyone?