Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Passing Global Properties to Hive Action Parameter

avatar
Rising Star

Hey guys,

I would like to pass properties defined in the <global> section by <job-xml> or <configuration> to a Hive Action parameter. But I cannot manage to do so. Example config:

<workflow>
<global>
<job-xml>hive_params.xml</job-xml>
</global>
...
<hive 0.6>
<script>h_script.hql</script>
<param>test=${test_from_hive_params_xml}</param>
</hive>

How can I achive this properly without changing my hive-site.xml? Or to pass a second xml to Hive but I do not want to change my hive-site.xml.

This is needed since I do not want to type all parameters by hand and pass it via the global section.

Looking forward to your help!

Thanks and regards

1 ACCEPTED SOLUTION

avatar
Super Guru

@Ken Jiiii

You could create another file and call it from h_script.hql, or you just add lines to h_script.hql like the following:

This set statements will override the global settings for that specific job session.

Example of line:

set mapred.reduce.tasks=32;

Almost anything that is Hive global environments XML is game for override at session level.

Look at this post: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer....

This shows a lot of these set statements which you can include in your set_h_script.hql invoked from h_script.hql or in the block added the beginning of your h_script.hql. Personally, I prefer a separate file invoked from my job script.

View solution in original post

1 REPLY 1

avatar
Super Guru

@Ken Jiiii

You could create another file and call it from h_script.hql, or you just add lines to h_script.hql like the following:

This set statements will override the global settings for that specific job session.

Example of line:

set mapred.reduce.tasks=32;

Almost anything that is Hive global environments XML is game for override at session level.

Look at this post: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer....

This shows a lot of these set statements which you can include in your set_h_script.hql invoked from h_script.hql or in the block added the beginning of your h_script.hql. Personally, I prefer a separate file invoked from my job script.