Support Questions

Find answers, ask questions, and share your expertise

how to deploy Customized Hive Avro serdes library

avatar
Explorer

I have written one Avro ser/des library for Hive to read/write Avro binary data file, which does not contain the Avro schema. And I copied this library into $HIVE/lib.

When I query the Hive external table, which uses my customized ser/des library, (e.g., select * from user_avro), it works as expected.

However if I insert data into that Hive external table like "INSERT OVERWRITE TABLE user_avro SELECT * FROM user), it complains that the my customized ser/des class cannot be found.

This output difference from Hive console is that the second one is launching MapReduce job.

Does anyone know where I need to copy this customized ser/des library except $HIVE/lib?

Many thanks in advance.

1 ACCEPTED SOLUTION

avatar
Super Guru

@yong yang apart from adding jars at each session level as suggested by @Jitendra Yadav you can add them permanently using hive.aux.jars.path

<property>
  <name>hive.aux.jars.path</name>
  <value>/var/lib/hive</value>
</property>

View solution in original post

3 REPLIES 3

avatar
Super Guru

@yong yang

Try using below command on hive shell before executing insert command.

hive>  add jar '\your jar file path';

avatar
Super Guru

@yong yang apart from adding jars at each session level as suggested by @Jitendra Yadav you can add them permanently using hive.aux.jars.path

<property>
  <name>hive.aux.jars.path</name>
  <value>/var/lib/hive</value>
</property>

avatar
Explorer

Thanks you very much for your reply. Based on your suggestions, I have added the following into hive-env.sh through Ambari Hive configuration interface:

export HIVE_AUX_JARS_PATH=<my jar full path>

and then restart the hive.

Now it works as expected in both Hive console and Hue BeesWax UI.

Many thanks again.