question Re: is it possible to expose dataset sample data(exp first 10 rows) using atlas? in Support Questions

is it possible to expose dataset sample data(exp first 10 rows) using atlas?

aymen_zaiter — Mon, 07 Jan 2019 20:01:20 GMT

Hello,

i am new to atlas and i am wondering if there is a solution making atlas expose, in addition to a dataset metadata, a sample data of this dataset ( 10 first rows for example).

for a purpose of data governance, the problem i am solving is that the metadata could not be so comprehensive functionally, and reading a sample data will make more sens to explain a dataset content.

if , actually, there is not a solution please help me find were should i invest my effort ( inner development within atlas, or treat the problem using a third party accessing hive,scoop or hbase directly to get the sample data)

thanks in advance.

Re: is it possible to expose dataset sample data(exp first 10 rows) using atlas?

wcbdata — Wed, 09 Jan 2019 23:10:18 GMT

There are two ways I'd suggest off the top of my head.

The simplest might be to add a metadata tag that would contain a small sample in CSV, JSON or HTML format, and populate it via the API or Atlas Kafka topics. For example, you could use HDF NiFi to periodically sample each table in the Hive, format the data, and populate the attribute.
For a more integrated approach, you might consider using the DataPlane DSS data profiler framework to add a "sample" profile that could be stored alongside the other profiler metadata and surfaced in DSS.

Re: is it possible to expose dataset sample data(exp first 10 rows) using atlas?

aymen_zaiter — Wed, 16 Jan 2019 23:24:13 GMT

Thank you for your reply, i think i would go for the first option making a metadata tag for each dataset.

Re: is it possible to expose dataset sample data(exp first 10 rows) using atlas?

wcbdata — Thu, 17 Jan 2019 00:23:33 GMT

Glad that helped! My colleague Greg Keys has a great series here on HCC about extending Atlas that may help you out, too:

https://community.hortonworks.com/articles/229421/customizing-atlas-part1-model-governance-traceabil.html

https://community.hortonworks.com/articles/231988/customizing-atlas-part2-deep-source-metadata-embed.html