Passing Global Properties to Hive Action Parameter

[#1]

Hey guys,

I would like to pass properties defined in the <global> section by <job-xml> or <configuration> to a Hive Action parameter. But I cannot manage to do so. Example config:

<workflow>
<global>
<job-xml>hive_params.xml</job-xml>
</global>
...
<hive 0.6>
<script>h_script.hql</script>
<param>test=${test_from_hive_params_xml}</param>
</hive>

How can I achive this properly without changing my hive-site.xml? Or to pass a second xml to Hive but I do not want to change my hive-site.xml.

This is needed since I do not want to type all parameters by hand and pass it via the global section.

Looking forward to your help!

Thanks and regards

[#2 · Accepted solution]

@Ken Jiiii

You could create another file and call it from h_script.hql, or you just add lines to h_script.hql like the following:

This set statements will override the global settings for that specific job session.

Example of line:

set mapred.reduce.tasks=32;

Almost anything that is Hive global environments XML is game for override at session level.

Look at this post: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer....

This shows a lot of these set statements which you can include in your set_h_script.hql invoked from h_script.hql or in the block added the beginning of your h_script.hql. Personally, I prefer a separate file invoked from my job script.