Support Questions
Find answers, ask questions, and share your expertise

Mapreduce and HCatalog Integration - HCatOutputFormat ClassNotFoundException

Contributor

I ran into issues while trying to read the hive table and write to a hive table using mapreduce.

Input and output:

Input hive table: mr_input_text

Output hive table: mr_output_text

Command that runs the MapReduce job

hadoop  jar mr-hcat.jar mr_input_text mr_output_text

Environment: HDP 2.3.2

This fails with the following exception. Used "yarn logs -applicationId " command to get this log.

java.lang.RuntimeException: java.lang.ClassNotFoundException :class org.apache.hcatalog.mapreduce.HCatOutputFormat not found

I tried setting the -libjars with the hcatalog core jar as follows, but still it fails.

hadoop  jar mr-hcat.jar mr_input_text mr_output_text -libjars /usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar

Note:

  1. I tried running from both as hdfs and hive user.
  2. Already looked at stackoverflow link
1 ACCEPTED SOLUTION

Mentor

have you looked at this page https://cwiki.apache.org/confluence/display/Hive/HCatalog+InputOutput

There are a bunch of dependencies you need to include

View solution in original post

2 REPLIES 2

Mentor

have you looked at this page https://cwiki.apache.org/confluence/display/Hive/HCatalog+InputOutput

There are a bunch of dependencies you need to include

Contributor

@Artem Ervits HCatOutputFormat class is in fact in the jar /usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar only. Actually its not about this specific class or jar to begin with; Its actually the command that i used; Changing it to the following works.

Note that the program arguments come at the end. The link that you suggested made me try this. Also, the other jars mentioned in the above link need to be added as it complained about other classes as well one by one. Those jars may be different based on the distribution and version.

In HDP 2.3.2 i did the following

export HCAT_HOME=/usr/hdp/current/hive-webhcat
export HIVE_HOME=/usr/hdp/current/hive-client


export LIB_JARS=$HCAT_HOME/share/hcatalog/hive-hcatalog-core.jar,$HIVE_HOME/lib/hive-metastore.jar,$HIVE_HOME/lib/libthrift-0.9.2.jar,$HIVE_HOME/lib/hive-exec.jar,$HIVE_HOME/lib/libfb303-0.9.2.jar,$HIVE_HOME/lib/jdo-api-3.0.1.jar,$HIVE_HOME/lib/datanucleus-api-jdo-3.2.6.jar


hadoop  jar mr-hcat.jar <mainclass> -libjars ${LIB_JARS} mr_input_text mr_output_text
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.