Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Any Tool/python library available for data profiling?

Solved Go to solution

Any Tool/python library available for data profiling?

Expert Contributor

Hi ,

I want to profile data which is present in sQL type databases(like hive/MySQL).

Does anyone know which tool is suitable for this?

It that tool is of hortonworks community then that will be best for me.

I have already tried pyxplorer python library but that didn't work for me because some installation problem.

please sent me tool link,Every where I am seeing Ta lend for data profiling but does talend provide all features related to data quality and data profiling(features like counting number of table,number of columns,distinct column values,min/max column value etc).

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Any Tool/python library available for data profiling?

Guru

For profiling data off Hadoop, see https://community.hortonworks.com/questions/35396/data-quality-analysis.html

For profiling data on Hadoop, the best solution for you should be:

  • zeppelin as your client/UI
  • spark in zeppelin as your toolset to profile

Both zeppelin and spark are extremely powerful tools for interacting with data and are packaged in HDP. Zeppelin is a browser-based notebook UI (like iPython/Jupyter) that excels at interacting with and exploring data. Spark of course is in-memory data analysis and is lightening fast. Both are key pieces in the future of Big Data analysis.

BTW, you can use python in spark or you can use scala, including integration of external libraries.

See the following links to get started:

http://hortonworks.com/apache/zeppelin/

http://www.social-3.com/solutions/personal_data_profiling.php

View solution in original post

1 REPLY 1
Highlighted

Re: Any Tool/python library available for data profiling?

Guru

For profiling data off Hadoop, see https://community.hortonworks.com/questions/35396/data-quality-analysis.html

For profiling data on Hadoop, the best solution for you should be:

  • zeppelin as your client/UI
  • spark in zeppelin as your toolset to profile

Both zeppelin and spark are extremely powerful tools for interacting with data and are packaged in HDP. Zeppelin is a browser-based notebook UI (like iPython/Jupyter) that excels at interacting with and exploring data. Spark of course is in-memory data analysis and is lightening fast. Both are key pieces in the future of Big Data analysis.

BTW, you can use python in spark or you can use scala, including integration of external libraries.

See the following links to get started:

http://hortonworks.com/apache/zeppelin/

http://www.social-3.com/solutions/personal_data_profiling.php

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here