Sentry HDFS ACL Sync with Hive HA Problems

[#1]

Since going to 5.13 I enabled Sentry HDFS ACL sync and also sentry HA, now that it's compatable with hive metastore ha.  Since doing this though i'm seeing lots of errors in the hive log, as well as canary failing to create databases and or partitions at random times.

Due to the metastore having problems, this impacts impala etc.

The hive error logs shows the following errors:

2017-12-15 14:39:13,093 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-5-thread-46]: Failed to sync requested HMS notifications up to the event ID: 819694
2017-12-15 14:39:13,103 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: failed to execute syncNotifications
2017-12-15 14:39:13,104 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: Thrift call failed
2017-12-15 14:40:53,180 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: failed to execute syncNotifications
2017-12-15 14:40:53,181 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: Thrift call failed
2017-12-15 14:40:53,220 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: failed to execute syncNotifications
2017-12-15 14:40:53,221 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: Thrift call failed
2017-12-15 14:42:33,111 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: failed to execute syncNotifications
2017-12-15 14:42:33,112 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-48]: Thrift call failed
2017-12-15 14:44:13,189 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: failed to execute syncNotifications
2017-12-15 14:44:13,189 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-49]: Thrift call failed
2017-12-15 14:44:13,225 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: failed to execute syncNotifications
2017-12-15 14:44:13,225 ERROR org.apache.sentry.core.common.transport.RetryClientInvocationHandler: [pool-5-thread-47]: Thrift call failed
2017-12-15 14:44:13,226 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-5-thread-47]: Failed to sync requested HMS notifications up to the event ID: 819785

And the sentry service is full of the following:

2017-12-15 14:45:53,199 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.
2017-12-15 14:46:18,056 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.
2017-12-15 14:47:33,093 ERROR org.apache.thrift.server.TThreadPoolServer: Thrift error occurred during processing of message.


2017-12-15 14:56:41,602 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:44,617 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:47,630 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:50,641 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!
2017-12-15 14:56:53,654 ERROR org.apache.sentry.service.thrift.HMSFollower: HMSFollower cannot connect to HMS!!

Any ideas anyone?

[#2]

I ran into a similar issue that was also logging this:

ERROR org.apache.sentry.service.thrift.HMSFollower: Failed to get the last processed notification id from sentry store, Skipping the processing
javax.jdo.JDODataStoreException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1

If you're seeing something similar to that, check your mysql-connector-java.jar version on all of your hosts, it may need updating.

[#3]

Did you solve this issue? I am facing the same issue.

[#4]

No, still outstanding.  Will try enabling again after upgrading to 5.14

[#5]

Hi all, @chriswalton007, @kmeansServiceNw, @georgewad,

I have had the same problem (always with the CDH 5.13.1) and at the moment seems work correctly after change the following parameter into Hive configuration service:

sentry.hdfs.sync.metastore.cache.fail

and put it in off.

Could you try this and let me know if it works also for you?

Many thanks in advance guys for the kind cooperation!

Regards,

Alex

[#6]

I also had the same problem last Friday afternoon. I did nothing on the weekend. When I come back this Monday, the problem went away for itself. Our cluster has about 400000 partitions. I think it may take a long time to finish the syncing process from hive to hdfs via senty. But I don't know for sure. Could anyone give me some explanation?

[#7]

I think I find the answer. It's the same as I guessed. See this page for more information. It says When you install Sentry, Sentry performs a full Hive metastore snapshot. This causes Hive metastore canary test failures while it synchronizes. Once the snapshot is complete, the canary test will stabilize.