@yukti 

You seems to be using incorrect version of "hive-exec" / "hive-metastore" JAR.

Can you please tell me from where did you get the 3.1.0.;0-78 version of JARs (as your Kafka jar version is"kafka-handler-3.1.0.3.1.0.0-78.jar" so looks like you might have downloaded that JAR from HDP 3.1 installation?  Is that correct?  If yes then can you also take the hive-metastore JAR of the same version and then try.

You can see the difference here between "hive-metastore-3.1.0.3.1.0.0-78.jar" and "hive-metastore-0.9.0.jar" JARs. From my HDP 3.1 installation.

# ls -l /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar
lrwxrwxrwx. 1 root root 35 Feb 22  2019 /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar -> hive-metastore-3.1.0.3.1.0.0-78.jar

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-metastore/lib/hive-standalone-metastore-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
Compiled from "DefaultHiveMetaHook.java"
public abstract class org.apache.hadoop.hive.metastore.DefaultHiveMetaHook implements org.apache.hadoop.hive.metastore.HiveMetaHook {
  public org.apache.hadoop.hive.metastore.DefaultHiveMetaHook();
  public abstract void commitInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
  public abstract void preInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
  public abstract void rollbackInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
}

.

On the other have we can see that "hive-metastore-0.9.0.jar" jar does not contains that class.

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /tmp/hive-metastore-0.9.0.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
Error: class not found: org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

.

If your question is answered then, Please make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.