04-09-2016 02:39 PM
environment: standalone mac mini running OS/X el capitan 10.11.2
i am attempting to import a database table into hive using sqoop. i see the data files in HDFS, but do not see the new table after running 'show tables' in hive. after i manually give the create table statement (which appears in the output of sqoop --verbose), then i can see the table and select from it.
i have hive configured to use derby as its metastore, using a well-known directory:
<description>JDBC connect string for a JDBC metastore</description>
i think the problem is that sqoop cannot interact with the hive metastore. every time i run the sqoop command:
sqoop import --verbose --connect jdbc:mysql://localhost/mylocaldb --username ..... --password ..... --table mytable --hive-import
it leaves a directory called metastore_db in the current directory. so i think the fix is to get sqoop to use the hive metastore, instead of doing whatever it's doing. setting HIVE_CONF_DIR to $HIVE_HOME/conf had no effect.
am i on the right track? how can i get this to work?
12-23-2016 10:17 AM
The solution that I kind of stumbled upon which worked for me is to put hive-site.xml in the folder where you are running the sqoop command. Similiar approach should work from oozie as well.
01-01-2017 04:46 PM
Sqoop will internally trigger Hive CLI command, so you will need to have proper hive-site.xml for Hive CLI to function. Can you please confirm that you can create tables when running Hive CLI by itself? (Not Beeline).
The Sqoop's --verbose output should give you some hint on whether table creation failed or succeeded, can you put the full output here for review?
05-03-2017 04:26 PM
I have the same issue but I am running my sqoop command as part of an oozie workflow in HUE. My job finishes successfully but the table does not get created under the database that I specify or under the default database. I can see the files that get imported but they are under my users default directory.
Any suggestions ?
05-04-2017 07:47 PM
Sadly I dont think the Oracle jdbc driver is in place for me to try that which is why I used HUE in the first place. I'll see if someone can install it for me.
05-04-2017 08:25 PM
Please also enabled verbose logging for sqoop action in Oozie, and get the full output of the sqoop command when running in Oozie, this can help us understand how the job was run inside Oozie anc compare it with when running by itself.