Mapreduce and HCatalog Integration - HCatOutputFormat ClassNotFoundException

[#1]

I ran into issues while trying to read the hive table and write to a hive table using mapreduce.

Input and output:

Input hive table: mr_input_text

Output hive table: mr_output_text

Command that runs the MapReduce job

hadoop  jar mr-hcat.jar mr_input_text mr_output_text

Environment: HDP 2.3.2

This fails with the following exception. Used "yarn logs -applicationId " command to get this log.

java.lang.RuntimeException: java.lang.ClassNotFoundException :class org.apache.hcatalog.mapreduce.HCatOutputFormat not found

I tried setting the -libjars with the hcatalog core jar as follows, but still it fails.

hadoop  jar mr-hcat.jar mr_input_text mr_output_text -libjars /usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar

Note:

1. I tried running from both as hdfs and hive user.
2. Already looked at stackoverflow link

[#2 · Accepted solution]

have you looked at this page https://cwiki.apache.org/confluence/display/Hive/HCatalog+InputOutput

There are a bunch of dependencies you need to include

[#3]

@Artem Ervits HCatOutputFormat class is in fact in the jar /usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar only. Actually its not about this specific class or jar to begin with; Its actually the command that i used; Changing it to the following works.

Note that the program arguments come at the end. The link that you suggested made me try this. Also, the other jars mentioned in the above link need to be added as it complained about other classes as well one by one. Those jars may be different based on the distribution and version.

In HDP 2.3.2 i did the following

export HCAT_HOME=/usr/hdp/current/hive-webhcat
export HIVE_HOME=/usr/hdp/current/hive-client


export LIB_JARS=$HCAT_HOME/share/hcatalog/hive-hcatalog-core.jar,$HIVE_HOME/lib/hive-metastore.jar,$HIVE_HOME/lib/libthrift-0.9.2.jar,$HIVE_HOME/lib/hive-exec.jar,$HIVE_HOME/lib/libfb303-0.9.2.jar,$HIVE_HOME/lib/jdo-api-3.0.1.jar,$HIVE_HOME/lib/datanucleus-api-jdo-3.2.6.jar


hadoop  jar mr-hcat.jar <mainclass> -libjars ${LIB_JARS} mr_input_text mr_output_text