Recommended data quality test suite for Hive / Pig / Oozie

[#1]

Hi

I am looking for Best practices around data quality Testing for hive / pig/ oozie based ETL.

Client is looking at tools like Data flux data quality for Hadoop .

If there are any alternate recommendations , please update this question.

[#2 · Accepted solution]

@pbalasundaram

https://www.trifacta.com/ - It can be used for data quality checks during cleansing

http://vis.stanford.edu/wrangler/ - open source version of trifacta

https://www.talend.com/resource/data-quality-tools.html

https://www.ataccama.com/products/big-data-platform-for-hadoop/big-data-engine

[#3]

Hi Neeraj - Trifacta seems to be a data wrangling tool, does it also provide data quality measures OOTB ?

[#4]

@pbalasundaram I have heard that it can be used for the quality while wrangling.

https://www.trifacta.com/wp-content/uploads/2014/01/Trifacta_DataTransformValue_WP.pdf

[#5]

@pbalasundaram are you still having issues with this? Can you accept best answer or provide your own solution?

[#6]

Hi Neeraj, for data quality testing is there a model script developed on pig or spark, rather than using a tool. Thanks.

[#7]

2020 Update, what are the preferred data quality tools compatible with CDH for Hive,Hbase and Solr? Our team is looking at Apache Griffin. 

Regards, 
Nithya Koka