question Mapreduce and HCatalog Integration - HCatOutputFormat ClassNotFoundException in Archives of Support Questions (Read Only)

Mapreduce and HCatalog Integration - HCatOutputFormat ClassNotFoundException

learninghuman — Thu, 17 Mar 2016 03:28:06 GMT

I ran into issues while trying to read the hive table and write to a hive table using mapreduce.

Input and output:

Input hive table: mr_input_text
Output hive table: mr_output_text

Command that runs the MapReduce job

hadoop  jar mr-hcat.jar mr_input_text mr_output_text

Environment: HDP 2.3.2

This fails with the following exception. Used "yarn logs -applicationId " command to get this log.

java.lang.RuntimeException: java.lang.ClassNotFoundException :class org.apache.hcatalog.mapreduce.HCatOutputFormat not found

I tried setting the -libjars with the hcatalog core jar as follows, but still it fails.

hadoop  jar mr-hcat.jar mr_input_text mr_output_text -libjars /usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar

Note:

I tried running from both as hdfs and hive user.
Already looked at stackoverflow link

Re: Mapreduce and HCatalog Integration - HCatOutputFormat ClassNotFoundException

aervits — Thu, 17 Mar 2016 11:18:20 GMT

have you looked at this page https://cwiki.apache.org/confluence/display/Hive/HCatalog+InputOutput

There are a bunch of dependencies you need to include

Re: Mapreduce and HCatalog Integration - HCatOutputFormat ClassNotFoundException

learninghuman — Thu, 17 Mar 2016 21:48:06 GMT

@Artem Ervits HCatOutputFormat class is in fact in the jar /usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar only. Actually its not about this specific class or jar to begin with; Its actually the command that i used; Changing it to the following works.

Note that the program arguments come at the end. The link that you suggested made me try this. Also, the other jars mentioned in the above link need to be added as it complained about other classes as well one by one. Those jars may be different based on the distribution and version.

In HDP 2.3.2 i did the following

export HCAT_HOME=/usr/hdp/current/hive-webhcat
export HIVE_HOME=/usr/hdp/current/hive-client


export LIB_JARS=$HCAT_HOME/share/hcatalog/hive-hcatalog-core.jar,$HIVE_HOME/lib/hive-metastore.jar,$HIVE_HOME/lib/libthrift-0.9.2.jar,$HIVE_HOME/lib/hive-exec.jar,$HIVE_HOME/lib/libfb303-0.9.2.jar,$HIVE_HOME/lib/jdo-api-3.0.1.jar,$HIVE_HOME/lib/datanucleus-api-jdo-3.2.6.jar


hadoop  jar mr-hcat.jar <mainclass> -libjars ${LIB_JARS} mr_input_text mr_output_text