Hive Major Compaction fails

[#1]

Hi,
I've been trying "ALTER TABLE (table_name) COMPACT 'MAJOR'" on my Hive3 environment, but it always fails. It seems that the merged base file is created but the delta is not deleted.
I found that it was because the HiveMetastore Client can't connect to the metastore because of GSS Initiate failed, but I can do other operation. Only the compaction fails.
How can I make compaction work? Here is the log:

2018-10-11T17:20:53,512 ERROR [Thread-18]: compactor.Cleaner (Cleaner.java:clean(317)) - Caught exception when cleaning, unable to complete cleaning of id:115189,dbname:(name_of_db),tableName:(name_of_table),partName:(partition_and_its_value),state:^@,type:MAJOR,properties:null,runAs:(user_who_has_the_file),tooManyAborts:false,highestWriteId:8 java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:301)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:169)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:279)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:401)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:397)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:352)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.access$000(Cleaner.java:64)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner$1.run(Cleaner.java:304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
... 2 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4299)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4367)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4347)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274)
... 13 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
... 22 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:51)
at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:48)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport.open(TUGIAssumingTransport.java:48)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:516)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:224)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:94)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4299)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4367)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4347)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:401)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:397)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:352)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.access$000(Cleaner.java:64)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner$1.run(Cleaner.java:304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:301)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:169)
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:565)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:224)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:94)
... 27 more

[#2]

@Kei Miyauchi You ever resolve this? I am running into the same issue and am not able to resolve it

[#3]

Same, Exactly the same issue for me. Any update ?

[#4]

Debug mode activated.

Logs on the metastore side when major compaction is activated. The process of cleaning fails (kerberos issue)

2019-08-25 21:42:51,729 INFO [Thread-24]: hive.metastore (HiveMetaStoreClient.java:open(451)) - Trying to connect to metastore with URI thrift://XXX:9083
2019-08-25 21:42:51,747 DEBUG [Thread-24]: security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1896)) - PrivilegedAction as:myuser (auth:PROXY) via hive/XXXX (auth:KERBEROS) from:org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
2019-08-25 21:42:51,748 DEBUG [Thread-24]: transport.TSaslTransport (TSaslTransport.java:open(261)) - opening transport org.apache.thrift.transport.TSaslClientTransport@58fdf6d4
2019-08-25 21:42:51,750 ERROR [Thread-24]: transport.TSaslTransport (TSaslTransport.java:open(315)) - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

[#5]

Here after a link related to Hive 3: http://www.adaltas.com/en/2019/07/25/hive-3-features-tips-tricks/

The author suggests the use of the hive user. So the hive keytab to execute merge query due to kerberos on my side. Is it the only way to make compaction work properly ?

I use Hive 2 (HDP 2.6.5) on my side but I face to the same issue.

[#6]

Something weird.
With the 'hive' used to insert data it works but instead of using this user to insert data, I've found that if the base directory (/apps/hive/warehouse/test.db/mytab/base_0000013) belongs to the 'hive' user, then the compaction succeeds (cleaning step is ok). Even if the delta_ directories don't belong to hive.
Weird, isn't it ?
Thus, this is a workaround but a little bit crappy.
But I'm open to any more suitable solution.