Support Questions
Find answers, ask questions, and share your expertise

Can Hive configuration be passed to Sqoop?

Contributor

Is there a way to pass Hive configuration to Sqoop that would be evaluated in the same fashion as '--hiveconf' when the actual Hive job is run to conduct the move task of data from landing to Hive table?

'-D' has no effect.

This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can Hive configuration be passed to Sqoop?

Mentor

The script will be executed by calling the installed copy of hive on the machine where Sqoop is run. If you have multiple Hive installations, or hive is not in your$PATH, use the --hive-home option to identify the Hive installation directory. Sqoop will use $HIVE_HOME/bin/hive from here.

View solution in original post

6 REPLIES 6

Re: Can Hive configuration be passed to Sqoop?

@kkane

"This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided"

Could you elaborate more ? Do you want Hive to not launch Tez container?

Re: Can Hive configuration be passed to Sqoop?

Contributor

Yes, I think that preventing the Tez container would fix the problems which are: Set the queue name and use delegation tokens for Sqoop when running in an Oozie shell.

The cluster default Hive execution engine is Tez.

Re: Can Hive configuration be passed to Sqoop?

Mentor

The script will be executed by calling the installed copy of hive on the machine where Sqoop is run. If you have multiple Hive installations, or hive is not in your$PATH, use the --hive-home option to identify the Hive installation directory. Sqoop will use $HIVE_HOME/bin/hive from here.

View solution in original post

Re: Can Hive configuration be passed to Sqoop?

Mentor

this is probably not the answer you're looking for but my guess is it should, especially with Oozie, you need to pass hive-site.xml. You probably need to do something like this

<command>[SQOOP-COMMAND]</command>
            <arg>[SQOOP-ARGUMENT]</arg>
	<file>lib/hive-site.xml</file>

Re: Can Hive configuration be passed to Sqoop?

Contributor

This is the Oozie shell vice Sqoop action but I took that and attempted anyway not thinking that Oozie configuration would not have effect on Sqoop running in the shell that calls its own Hive setup but it did. So, I used the dist cache <files></files> like you did in the Sqoop action and it worked.

Thanks.

Re: Can Hive configuration be passed to Sqoop?

Mentor

great, I think it will be worth if you show screenshots in this thread for everyone else to use.