Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

how to deploy Customized Hive Avro serdes library

Explorer

I have written one Avro ser/des library for Hive to read/write Avro binary data file, which does not contain the Avro schema. And I copied this library into $HIVE/lib.

When I query the Hive external table, which uses my customized ser/des library, (e.g., select * from user_avro), it works as expected.

However if I insert data into that Hive external table like "INSERT OVERWRITE TABLE user_avro SELECT * FROM user), it complains that the my customized ser/des class cannot be found.

This output difference from Hive console is that the second one is launching MapReduce job.

Does anyone know where I need to copy this customized ser/des library except $HIVE/lib?

Many thanks in advance.

1 ACCEPTED SOLUTION

@yong yang apart from adding jars at each session level as suggested by @Jitendra Yadav you can add them permanently using hive.aux.jars.path

<property>
  <name>hive.aux.jars.path</name>
  <value>/var/lib/hive</value>
</property>

View solution in original post

3 REPLIES 3

@yong yang

Try using below command on hive shell before executing insert command.

hive>  add jar '\your jar file path';

@yong yang apart from adding jars at each session level as suggested by @Jitendra Yadav you can add them permanently using hive.aux.jars.path

<property>
  <name>hive.aux.jars.path</name>
  <value>/var/lib/hive</value>
</property>

Explorer

Thanks you very much for your reply. Based on your suggestions, I have added the following into hive-env.sh through Ambari Hive configuration interface:

export HIVE_AUX_JARS_PATH=<my jar full path>

and then restart the hive.

Now it works as expected in both Hive console and Hue BeesWax UI.

Many thanks again.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.