Hi, I would like to get rid of some duplicates in my Hive table. The duplicates differs by the variable DateExtraction. DateExtraction is a date formated as "yyyy/MM/dd". In case of duplicate i would like to keep the most recent observation. Any clues are most welcome 🙂 Here is the query i used to create my table: DROP TABLE IF EXISTS netodb.oneTable;
CREATE EXTERNAL TABLE IF NOT EXISTS netodb.oneTable(
Entreprise string,Industrie string,Source string,Type string,Auteur string,
Id string,Texte string,DateCreation string,Likes bigint,Partages bigint,
ObjetParent string,Users_mentionned string,Hashtags_uses string,View_count string,
Dislike_count string,Comment_count string,DateExtraction date)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY "|"
LOCATION "/tmp/NetoDB/OneTable"
TBLPROPERTIES("transactional"="true");
... View more
