question how to deploy Customized Hive Avro serdes library in Support Questions

how to deploy Customized Hive Avro serdes library

donaldyy — Tue, 31 May 2016 19:48:46 GMT

I have written one Avro ser/des library for Hive to read/write Avro binary data file, which does not contain the Avro schema. And I copied this library into $HIVE/lib.

When I query the Hive external table, which uses my customized ser/des library, (e.g., select * from user_avro), it works as expected.

However if I insert data into that Hive external table like "INSERT OVERWRITE TABLE user_avro SELECT * FROM user), it complains that the my customized ser/des class cannot be found.

This output difference from Hive console is that the second one is launching MapReduce job.

Does anyone know where I need to copy this customized ser/des library except $HIVE/lib?

Many thanks in advance.

Re: how to deploy Customized Hive Avro serdes library

jyadav — Tue, 31 May 2016 20:10:58 GMT

@yong yang

Try using below command on hive shell before executing insert command.

hive>  add jar '\your jar file path';

Re: how to deploy Customized Hive Avro serdes library

rajkumar_singh — Tue, 31 May 2016 20:35:16 GMT

@yong yang apart from adding jars at each session level as suggested by @Jitendra Yadav you can add them permanently using hive.aux.jars.path

<property>
  <name>hive.aux.jars.path</name>
  <value>/var/lib/hive</value>
</property>

Re: how to deploy Customized Hive Avro serdes library

donaldyy — Tue, 31 May 2016 22:31:20 GMT

Thanks you very much for your reply. Based on your suggestions, I have added the following into hive-env.sh through Ambari Hive configuration interface:

export HIVE_AUX_JARS_PATH=<my jar full path>

and then restart the hive.

Now it works as expected in both Hive console and Hue BeesWax UI.

Many thanks again.