Created 07-02-2019 04:51 PM
I imported Oracle table using Hcatalog to import as ORC with below Syntax. In every run it appends data instead of overwrite. Is this any options to avoid this append ?
sqoop import \
--connect "jdbc:oracle:thin:@${ORACLE_TNS_STRING}" \
--username $USER \
--password $PASSWD \
--table tab1 \
--hcatalog-database hive_db \
--hcatalog-table hive_table \
--hcatalog-storage-stanza "stored as orc tblproperties ("orc.compress"="SNAPPY")" \
--m 1
Created 07-03-2019 05:23 AM
Create a shell script, that will truncate your hive orc table and then sqoop import to hive orc table.
Sample shell script:-
1.For Managed(internal) Tables:
bash$ cat sq.sh #truncate the hive table first hive -e "truncate table default.my_table_orc" #sqoop import into Hive table sqoop import --connect jdbc:mysql://localhost:3306/<db_name> --username root --password "<password>" --table <table_name> --hcatalog-database <hive_database> --hcatalog-table <hive_table_name> --hcatalog-storage-stanza "stored as orcfile"
2.For External Tables:
bash$ cat sq.sh #drop the hive table first hive -e "drop table default.my_table_orc" #sqoop import into Hive table sqoop import --connect jdbc:mysql://localhost:3306/<db_name> --username root --password "<password>" --table order_items --hcatalog-database <hive_database> --hcatalog-table <hive_table_name> --create-hcatalog-table --hcatalog-storage-stanza "stored as orcfile"