How to deal with duplicate values while importing data from RDBMS to HDFS?

[#1]

I am using ambari.

sqoop import --connect "jdbc:sqlserver://localhost:1433;database=db_name;username=user_name;pasword= password" --table table_name --target-dir /Sqoop/output --append --incremental append --check-column ID --last-value 100

while executing this command, it loading newly added rows in the existing directory but it shows duplicated records also.

how to avoid these duplicated records using sqoop?

[#2 · Accepted solution]

Below posts might help:

https://community.hortonworks.com/questions/63761/sqoop-import-to-hive-again-stroing-repeted-recorde...

https://community.hortonworks.com/questions/51508/sqoop-imported-more-records-than-source.html