question Re: sqoop import to hive again stroing repeted recordes in same hive table in Archives of Support Questions (Read Only)

sqoop import to hive again stroing repeted recordes in same hive table

Former Member — Thu, 27 Oct 2016 17:28:54 GMT

sqoop import --connect jdbc:mysql://locahost/test --table a2 --username root --password -m 1 --hive-import --hive-database default --hive-table a2 --target-dir /tmp/n11 --driver com.mysql.jdbc.Driver 1.

The mysql table a2 contains 2 records.

example

id name

1 aa

2 bb

2. Initially i run the below query it create and load the 2 records to the hive

3.Then i run the same query again it stores the same records like this

id name

1 aa

2 bb

1 aa

2 bb

how avoid this duplicate records generation in hive table using sqoop please suggest me

please help me to solve this problem

thanks in advance

swathi

Re: sqoop import to hive again stroing repeted recordes in same hive table

gkeys — Fri, 28 Oct 2016 02:47:41 GMT

You must have a table with a column updating the modifed date (date of insert or update). If you have this column, then use: --check-column {modified_date_col} --incremental {lastmodified} --last-value {modified_date}. If you do not have this column, you cannot avoid your issue.

Re: sqoop import to hive again stroing repeted recordes in same hive table

Former Member — Fri, 28 Oct 2016 11:54:53 GMT

Thanku so much

Re: sqoop import to hive again stroing repeted recordes in same hive table

HadoopHelp — Fri, 17 May 2019 23:03:20 GMT

Dear All.

I have table in sql server that column contain random unique number there is no any primary key but we want to perform incremental append or lastmodified operation using sqoop so please help me.

Note:-This is Critical Issue.

Re: sqoop import to hive again stroing repeted recordes in same hive table

kushalbohra — Sat, 18 May 2019 04:12:54 GMT

Greg's answer applies to you as well for incremental import/export operation. Also if you have some column in your source table which is an sequential index etc then you can be used for --split-by clause for distribution of data per mapper to scale parallelism and reduce runtime of app.

My understanding is random numbers in a column if used for split key , can cause skew as well leading to different runtimes for map tasks.