@Manoj Dhake Hi, Atlas and Falcon serve very different purposes, but there are some areas where they touch base. Maybe that is where your confusion comes from. Atlas: -really like an 'atlas' to almost all of the metadata that is around in HDP like Hive metastore, Falcon repo, Kafka topics, Hbase table etc. This single view on metadata makes for some powerfull searching capabilities on top of that with full text search (based on solr) -Since Atlas has this comprehensive view on metadata it is also capable of providing insight in lineage, so it can tell by combining Hive DDL's what table was the source for another table. -Another core feature is that you assign tags to all metadata entities on Atlas. So you can say that column B in Hive table Y holds sensitive data by assigning a 'PII' tag to it. But a hdfs folder can also be assigned a 'PII' tag or a CF from Hbase. From there you can create tag based policies from Ranger to manage access to anything 'PII' tagged in Atlas. Falcon: -more like a scheduling and execution engine for HDP components like Hive, Spark, hdfs distcp, Sqoop to move data around and/or process data along the way. In a way Falcon is a much improved Oozie. -metadata of Falcon dataflows is actually sinked to Atlas through Kafka topics so Atlas knows about Falcon metadata too and Atlas can include Falcon processes and its resulting meta objects (tables, hdfs folders, flows) into its lineage graphs. I know that in the docs both tools claim the term 'data governance', but I feel Atlas is more about that then Falcon is. It is not that clear what Data Governance actually is. With Atlas you can really apply governance by collecting all metadata querying and tagging it and Falcon can maybe execute processes that evolve around that by moving data from one place to another (and yes, Falcon moving a dataset from an analysis cluster to an archiving cluster is also about data governance/management) Hope that helps ... View more

For profiling data off Hadoop, see https://community.hortonworks.com/questions/35396/data-quality-analysis.html For profiling data on Hadoop, the best solution for you should be: zeppelin as your client/UI spark in zeppelin as your toolset to profile Both zeppelin and spark are extremely powerful tools for interacting with data and are packaged in HDP. Zeppelin is a browser-based notebook UI (like iPython/Jupyter) that excels at interacting with and exploring data. Spark of course is in-memory data analysis and is lightening fast. Both are key pieces in the future of Big Data analysis. BTW, you can use python in spark or you can use scala, including integration of external libraries. See the following links to get started: http://hortonworks.com/apache/zeppelin/ http://www.social-3.com/solutions/personal_data_profiling.php ... View more

@Manoj Dhake When you create a new Trait/Tag in the UI are are actually creating a new type. When you associate it with an entity you are creating an instance of the Trait as a STRUCT type that only exists in the context of that entity. As you know, you can delete instances of a tag that is assigned to an entity. However, to delete the tag so that it does not show up in Atlas UI you would need to delete the type. At the moment, types cannot be deleted since there may be entities or even other types that are dependent on them. You probably can delete the trait directly from HBase, however, you may also compromise the integrity of the entire Atlas system. You would also have to clear any dependents that are indexed in Solr or graphed in Titan. I am sure these are all issues that the community will have to solve in dealing implemeting the JIRA you referenced. To answer the follow-up question you asked in the comments sections, you can get the instance of a trait only from the entity where the trait is associated as follows: curl -u admin:admin -X GET http://sandbox.hortonworks.com:21000/api/atlas/entities/fadaca14-7e58-4a2e-b04d-95a3010ce45b/traits {"requestId":"qtp921483514-312 - bb0f1276-c8fc-499b-ab5d-0e4fde11c796","results":["PII"],"count":1}[root@sandbox ~]# As you can see, this entity has a tag called PII and it does not have a GUID because it is STRUCT based on the PII type that represents PII trait. You can easily delete the STRUCT that is associated with an entity through the UI or REST api but there is now way to remove the type from which the trait is spawned for the reasons listed above. ... View more