@Sunile Manjee The dependencies are derived based on the entity description, once you create those entities using Falcon (UI or CLI). So for e.g., you define your cluster in the cluster entity xml, you specify the name.. <cluster colo="location1" description="primaryDemoCluster" name="primaryCluster" xmlns="uri:falcon:cluster:0.1"> When you define this cluster in a feed entity, the dependency gets created when you create the feed entity.. <feed description="Demo Input Data" name="demoEventData" xmlns="uri:falcon:feed:0.1"> <tags>externalSystem=eventData,classification=clinicalResearch</tags> <groups>events</groups> <frequency>minutes(3)</frequency> <timezone>GMT+00:00</timezone> <late-arrival cut-off="hours(4)"/> <clusters> <cluster name="primaryCluster" type="source"> <validity start="2015-08-10T08:00Z" end="2016-02-08T22:00Z"/> <retention limit="days(5)" action="delete"/> </cluster> </clusters> The same concept applies to processes to feed dependencies.. Take a look at this example for working set of falcon entities - https://github.com/sainib/hadoop-data-pipeline/tree/master/falcon ... View more

@Alex Raj So it appears your calling a Shell action which is expected to produce some output (within file system or hdfs) and you want to see that, is that correct? Or Are you actually wanting to capture the output (echo statements) in the script for the purpose of referencing those values in subsequent steps in the Oozie workflow? If its the latter, see the response from @Benjamin Leonhardi If its the former, which I believe you are asking then the answer is (you wont be thrilled) - It depends. It depends on what the script is doing. I can imagine few scenarios and will talk through that but let us know if you are doing something different in which case, we can talk specific about that. So here is what you MAY be doing in the script - writing to a local file with absolute path writing to a local file with relative path writing to a HDFS file with absolute path Writing to a local file with absolute path - Lets say the script does this - touch /tmp/a.txt In this case, the output gets created on the local filesystem of nodemanager where the task got executed. There is really no way to tell which one.. so you would have to check all nodes. The good thing is that you know what the absolute path is. Writing to a local file with relative path - Lets say the script does this - touch ./a.txt In this case, the output gets created on the local filesystem of nodemanager, where the task got executed, but relative to the working temp directory where workflow temporary files are created. There is really no way to tell which note and we may never even see the actual file because usually the temporary files are cleaned up after the workflow is executed. SO if the file is within the subdirectory then it will most likely be deleted. Writing to a HDFS file with absolute path <- This is the best way to setup the program because you know where to look for output. Lets say the script does this - echo "my content" >> /tmp/a.txt hdfs dfs -put /tmp/a.txt /tmp/a.txt In this case, the output gets created on HDFS & you know the path. So its easy to find. If you are not following the last approach, I would recommend that. Hope this helps. ... View more

@khushi kalra Like Neeraj and Artem pointed out, Apache Atlas is the right tool for managing metadata for Hadoop. Falcon is more for managing the data pipeline and data workflow management which is big part of overall data governance but not metadata. In addition to the links and resources provided, here is a Apache Atlas presentation video by Governance product manager, Andrew Ahn.. https://www.youtube.com/watch?v=LZR4qhKJeSI ... View more