Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can Hive configuration be passed to Sqoop?

avatar
Rising Star

Is there a way to pass Hive configuration to Sqoop that would be evaluated in the same fashion as '--hiveconf' when the actual Hive job is run to conduct the move task of data from landing to Hive table?

'-D' has no effect.

This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided.

1 ACCEPTED SOLUTION

avatar
Master Mentor

The script will be executed by calling the installed copy of hive on the machine where Sqoop is run. If you have multiple Hive installations, or hive is not in your$PATH, use the --hive-home option to identify the Hive installation directory. Sqoop will use $HIVE_HOME/bin/hive from here.

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@kkane

"This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided"

Could you elaborate more ? Do you want Hive to not launch Tez container?

avatar
Rising Star

Yes, I think that preventing the Tez container would fix the problems which are: Set the queue name and use delegation tokens for Sqoop when running in an Oozie shell.

The cluster default Hive execution engine is Tez.

avatar
Master Mentor

The script will be executed by calling the installed copy of hive on the machine where Sqoop is run. If you have multiple Hive installations, or hive is not in your$PATH, use the --hive-home option to identify the Hive installation directory. Sqoop will use $HIVE_HOME/bin/hive from here.

avatar
Master Mentor

this is probably not the answer you're looking for but my guess is it should, especially with Oozie, you need to pass hive-site.xml. You probably need to do something like this

<command>[SQOOP-COMMAND]</command>
            <arg>[SQOOP-ARGUMENT]</arg>
	<file>lib/hive-site.xml</file>

avatar
Rising Star

This is the Oozie shell vice Sqoop action but I took that and attempted anyway not thinking that Oozie configuration would not have effect on Sqoop running in the shell that calls its own Hive setup but it did. So, I used the dist cache <files></files> like you did in the Sqoop action and it worked.

Thanks.

avatar
Master Mentor

great, I think it will be worth if you show screenshots in this thread for everyone else to use.