https://community.cloudera.com/t5/Support-Questions/How-one-should-handle-de-duplication-of-data/m-p/120016#M82794 <P> <A rel="user" href="https://community.cloudera.com/users/9842/mpandit.html" nodeid="9842">@milind pandit</A> loaded question. First you have to define what the unique entity is. once that solved then you can use various tools like pig to parse through data and provide you single record. This can also be done via hive by using group by statement on your natural key to provide you single record from source. Lastly you can use tools like information or talend to do the same.</P> Fri, 26 Aug 2016 03:16:38 GMT sunile_manjee 2016-08-26T03:16:38Z https://community.cloudera.com/t5/Support-Questions/How-one-should-handle-de-duplication-of-data/m-p/120015#M82793 Fri, 26 Aug 2016 02:13:01 GMT https://community.cloudera.com/t5/Support-Questions/How-one-should-handle-de-duplication-of-data/m-p/120015#M82793 mpandit 2016-08-26T02:13:01Z https://community.cloudera.com/t5/Support-Questions/How-one-should-handle-de-duplication-of-data/m-p/120016#M82794 <P> <A rel="user" href="https://community.cloudera.com/users/9842/mpandit.html" nodeid="9842">@milind pandit</A> loaded question. First you have to define what the unique entity is. once that solved then you can use various tools like pig to parse through data and provide you single record. This can also be done via hive by using group by statement on your natural key to provide you single record from source. Lastly you can use tools like information or talend to do the same.</P> Fri, 26 Aug 2016 03:16:38 GMT https://community.cloudera.com/t5/Support-Questions/How-one-should-handle-de-duplication-of-data/m-p/120016#M82794 sunile_manjee 2016-08-26T03:16:38Z