question Re: Can Hive configuration be passed to Sqoop? in Archives of Support Questions (Read Only)

Can Hive configuration be passed to Sqoop?

kkane — Mon, 23 Nov 2015 23:34:52 GMT

Is there a way to pass Hive configuration to Sqoop that would be evaluated in the same fashion as '--hiveconf' when the actual Hive job is run to conduct the move task of data from landing to Hive table?

'-D' has no effect.

This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided.

Re: Can Hive configuration be passed to Sqoop?

nsabharwal — Mon, 23 Nov 2015 23:46:25 GMT

@kkane

"This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided"

Could you elaborate more ? Do you want Hive to not launch Tez container?

Re: Can Hive configuration be passed to Sqoop?

kkane — Tue, 24 Nov 2015 00:01:37 GMT

Yes, I think that preventing the Tez container would fix the problems which are: Set the queue name and use delegation tokens for Sqoop when running in an Oozie shell.

The cluster default Hive execution engine is Tez.

Re: Can Hive configuration be passed to Sqoop?

aervits — Tue, 24 Nov 2015 00:17:03 GMT

The script will be executed by calling the installed copy of hive on the machine where Sqoop is run. If you have multiple Hive installations, or hive is not in your$PATH, use the --hive-home option to identify the Hive installation directory. Sqoop will use $HIVE_HOME/bin/hive from here.

Re: Can Hive configuration be passed to Sqoop?

aervits — Tue, 24 Nov 2015 00:17:04 GMT

this is probably not the answer you're looking for but my guess is it should, especially with Oozie, you need to pass hive-site.xml. You probably need to do something like this

<command>[SQOOP-COMMAND]</command>
            <arg>[SQOOP-ARGUMENT]</arg>
	<file>lib/hive-site.xml</file>

Re: Can Hive configuration be passed to Sqoop?

kkane — Tue, 24 Nov 2015 02:56:56 GMT

This is the Oozie shell vice Sqoop action but I took that and attempted anyway not thinking that Oozie configuration would not have effect on Sqoop running in the shell that calls its own Hive setup but it did. So, I used the dist cache <files></files> like you did in the Sqoop action and it worked.

Thanks.

Re: Can Hive configuration be passed to Sqoop?

aervits — Tue, 24 Nov 2015 02:58:59 GMT

great, I think it will be worth if you show screenshots in this thread for everyone else to use.