@pbarna I worked on this as part of Atlas project. I realized that solution is not as simple given the numerous dependencies involved. Can you please tell me the version of Titan and Hadoop you are using? I attempted a similar exercise for using Titan 0.5.4 and Hadoop 2.6.3. My problem was to initiate Titan Index Repair job. This facility is built-in to Titan API. It uses MapReduce to initiate the repair. With some help, I realized that adding properties to yarn-site.xml and hbase-site.xml actually help. When you do update the properties in these files, be sure to use the <final>true</final> so that your settings override the default and take effect. Example: <property> <name>mapreduce.local.map.tasks.maximum</name> <value>10</value> <final>true</final> </property> Due to various reason I ended up writing a groovy script to achieve this. I can get into details if you are interested. My script is here. Please feel free to reach out if you think this was useful. Thanks @Nixon Rodrigues for letting me know about this question. ... View more

@Arsalan Siddiqi Thanks the excellent question. Your observations are valid. While Atlas does help with meeting compliance requirements, it is only part of the solution. To use traffic analogy, Atlas is the map (hence the name) and does not deal the cars on the road (traffic). To complete the picture, there needs to be some monitoring on what data gets ingested in the system and if all the data conforms with the norms setup. Please take a look at this presentation from Data Summit 2017. It explains how a system can be setup which helps with governance (realm of Atlas) and then also helps with spotting errors within the data itself. To summarize, to be able to spot errors with flow of data itself, you would need some other mechanism. Atlas will not help you in that respect. About your 2nd question: Atlas consumes notifications from Kafka by spawning a single thread and processing 1 notification at a time (see NotificationHookConsumer.java & AtlasKafkaConsumer.java). In case of systems with high throughput, the notifications will be queued with Kafa and you will see a lag in consumption of notifications. Kafka guarantees durability of messages. Atlas ensures that it consumes every message produced by Kafka. If messages are dropped for some reason, you would see them in Atlas' logs. We also test Atlas in high available scenarios. Also, to address the notification message question, I would urge you to use Atlas V2 client APIs (both on master and branch-0.8). Kafka does not mandate any message format since all it understands is bytes, so that should not be a determining criteria for choosing the client API version. I know this is a lot of text, I hope it helps. Please feel free to reach out if you need clarifications. ... View more

@David Miller DSL query should help you with this. Each hive_table has hive_db name in its property. This DSL query should help you: hive_table where (db.name like '*_final' or db.name like '*_temp') About filtering deleted entities in DSL, there isn't a way to do it yet. We are in process of improving DSL. As for the documentation, I agree, it needs improvement. There is no firm ETA on this yet. Given the current state, my suggestion would be to use basic query as much as possible. Hope this helps. ... View more

@Arsalan Siddiqi Your observations are accurate. In fact there is one initiative in progress that is to address this. Only that I don't have an ETA on when it will get done. ... View more

@subash sharma Have taken a look at the Hive Hook code? This should give you a good idea of the approach used. I don't know if that will be sufficient for all your needs, but it should give you a good idea on how to approach the problem. Hope this helps. ... View more

Let me attempt to answer all your questions: 1. Think of composite as composition in object oriented design world. An entity is part of the dependent entity. The lifetime of the child (contained entity) is determined by its parent. Child cannot exist without the parent. 2. I don't think there is a place where all of this is defined. We should definitely improve on our documentation. Here's link to code where this is defined. It is not ideal but better than nothing. 3. Relationships is a new concept. This implementation is only is master now. There is backward compatibility. Here's what the concept is: So far (before relationships) we would define associations between 2 entities as containment just by referencing one entity into another. See hive model in branch 0.8. Relationships merely allow you to capture this more comprehensively by modeling it separately. This will be apparent if you simply compare the hive model in master and branch-0.8. 4. For defining entities with composition, you could useAtlasEntity.AtlasEntityWithExtInfo. Add parent entity to the constructor and define the entities referenced as AtlasObjectIds. The creation API will take care of resolving the references. I am attaching JSON for reference. CURL entity creation is also added for reference. Also see these object diagrams. Hope this helps. @Sarath Subramanian Thanks for your help in drafting this reply. hive-table-sample.zipcreate-hive-table-entity.txt ... View more