How to see the Data Provenance and Lineage in Data Flow on Public Cloud?

Question

This video (timestamped) shows you can list the queue on connections, and see provenance and lineage in flow designer: https://youtu.be/8cZJ9CyLYyI?t=5904 But in the public cloud version of Cloudera Data Flow, that functionality is missing. I can list queue and see data in many formats, but no provenance and lineage. Do we need Data Hub to do this or am I missing something? I am doing this from the Web console of Data Flow Designer.

I am trying this out on a trial account of the Cloudera DataFlow, so not sure if that affects my experience. Provenance and replay was an important function I wanted to try.

Screenshots (Imgur link)

mburgess · Accepted Answer

Provenance/lineage is not currently visible from the Flow Designer. This is intended because the Flow Designer UI is for flow design regardless of whether there is a Test Session or deployment active. Provenance and lineage is associated with actual data running through a deployment, so to view these you'll need to navigate to the Cloudera Flow Management (NiFi) canvas from the deployment view once your flow has been deployed. From the canvas you can proceed as the video instructs and hopefully it looks familiar at that point.

DianaTorres · Answer

@mansmaan Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @SAMSAL @mburgess who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.