Problem was seen when the client was running Metastore HA and both configured with org.apache.hadoop.hive.ql.txn.compactor.Initiator. this is what we observed from the logs of both metastore service
// metastore 1
ERROR compactor.Worker (Worker.java:run(181)) - Caught an exception in the main loop of compactor worker lnxhdpap02.smrcy.com-33, MetaException(message:Unable to connect to transaction database com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
// metastore 2
ERROR txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, Deadlock found when trying to get lock; try restarting transaction
org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs inside the metastore service to manage compactions of ACID tables. There should be exactly 1 instance of this thread (even with multiple Thrift services).
set “hive.compactor.initiator.on" only for single instance of metastore service.