Created 11-23-2015 03:34 PM
Is there a way to pass Hive configuration to Sqoop that would be evaluated in the same fashion as '--hiveconf' when the actual Hive job is run to conduct the move task of data from landing to Hive table?
'-D' has no effect.
This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided.
Created 11-23-2015 04:17 PM
Want to get a detailed solution you have to login/registered on the community
Register/LoginCreated 11-23-2015 03:46 PM
"This is particularly needed when Hive on Tez is the default engine and Hive starts up a Tez container no matter what other configuration has been provided"
Could you elaborate more ? Do you want Hive to not launch Tez container?
Created 11-23-2015 04:01 PM
Yes, I think that preventing the Tez container would fix the problems which are: Set the queue name and use delegation tokens for Sqoop when running in an Oozie shell.
The cluster default Hive execution engine is Tez.
Created 11-23-2015 04:17 PM
Want to get a detailed solution you have to login/registered on the community
Register/LoginCreated 11-23-2015 04:17 PM
this is probably not the answer you're looking for but my guess is it should, especially with Oozie, you need to pass hive-site.xml. You probably need to do something like this
<command>[SQOOP-COMMAND]</command> <arg>[SQOOP-ARGUMENT]</arg> <file>lib/hive-site.xml</file>
Created 11-23-2015 06:56 PM
This is the Oozie shell vice Sqoop action but I took that and attempted anyway not thinking that Oozie configuration would not have effect on Sqoop running in the shell that calls its own Hive setup but it did. So, I used the dist cache <files></files> like you did in the Sqoop action and it worked.
Thanks.
Created 11-23-2015 06:58 PM
great, I think it will be worth if you show screenshots in this thread for everyone else to use.