@Tim Mitchell can you confirm the contents of the following directory: [root@localhost atlas-hbase-plugin-impl]# ls -lt /usr/hdp/current/atlas-server/hook/hbase/atlas-hbase-plugin-impl total 17664 -rw-r--r--. 1 atlas hadoop 46464 Dec 6 04:32 hbase-bridge-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 13204 Dec 6 04:31 hdfs-model-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 19107 Dec 6 04:27 atlas-client-v2-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 61496 Dec 6 04:27 atlas-notification-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 7322236 Dec 6 04:27 kafka_2.11-2.0.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 1894753 Dec 6 04:27 kafka-clients-2.0.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 37544 Dec 6 04:27 atlas-client-common-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 69562 Dec 6 04:27 atlas-common-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 465346 Dec 6 04:26 atlas-intg-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 45927 Nov 13 10:54 jsr311-api-1.1.jar -rw-r--r--. 1 atlas hadoop 5749423 Nov 13 10:49 scala-library-2.11.12.jar -rw-r--r--. 1 atlas hadoop 66981 Nov 13 10:25 jackson-annotations-2.9.6.jar -rw-r--r--. 1 atlas hadoop 323848 Nov 13 10:25 jackson-core-2.9.6.jar -rw-r--r--. 1 atlas hadoop 1349339 Nov 13 10:25 jackson-databind-2.9.6.jar -rw-r--r--. 1 atlas hadoop 53275 Oct 22 2016 jersey-multipart-1.19.jar -rw-r--r--. 1 atlas hadoop 165345 Oct 21 2016 jersey-json-1.19.jar -rw-r--r--. 1 atlas hadoop 362679 Oct 21 2016 commons-configuration-1.10.jar You should have the listed jars for HBase hook to work correctly. ... View more

@Karthikeyan Arjunan, sqoop hook is the way to import RDBMS data into Atlas. Currently we don't have hooks specific to RDBMS db like Oracle or Teradata ... View more

currently there is no way to disable lineage rendering in UI for any table. ... View more

Relationship category determines the style of relationship around containment and lifecycle. UML terminology is used for the values. ASSOCIATION is a relationship with no containment. COMPOSITION and AGGREGATION are containment relationships. The difference being in the lifecycles of the container and its children. In the COMPOSITION case, the children cannot exist without the container. For AGGREGATION, the life cycles of the container and children are totally independent. ... View more

@Owez Mujawar you should have the following libraries for import-hive.sh to work. Not sure why you have lot of libraries. Did you manually copy atlas lib jars here?/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl [root@localhost atlas-hive-plugin-impl]# ls -ltr total 37448 -rw-r--r--. 1 atlas hadoop 4573750 May 11 2018 scala-reflect-2.11.8.jar -rw-r--r--. 1 atlas hadoop 802818 May 11 2018 scalap-2.11.8.jar -rw-r--r--. 1 atlas hadoop 5744974 May 11 2018 scala-library-2.11.8.jar -rw-r--r--. 1 atlas hadoop 15487351 May 11 2018 scala-compiler-2.11.8.jar -rw-r--r--. 1 atlas hadoop 1590487 May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 7532355 May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 67829 May 11 2018 json4s-native_2.11-3.2.11.jar -rw-r--r--. 1 atlas hadoop 589462 May 11 2018 json4s-core_2.11-3.2.11.jar -rw-r--r--. 1 atlas hadoop 82421 May 11 2018 json4s-ast_2.11-3.2.11.jar -rw-r--r--. 1 atlas hadoop 53275 May 11 2018 jersey-multipart-1.19.jar -rw-r--r--. 1 atlas hadoop 70540 May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 11639 May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 577458 May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 232482 May 11 2018 gson-2.5.jar -rw-r--r--. 1 atlas hadoop 408746 May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 98221 May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 222090 May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 64778 May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 18040 May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 43414 May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 35271 May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar ... View more

@Owez Mujawar So you see the same exception when you try to publish a sample message to ATLAS_ENTITIES topic as atlas user. Some kafka config has been messed up - kafka brokers have been added or removed. Please follow https://community.hortonworks.com/content/supportkb/150063/how-to-add-new-kafka-brokers-to-a-cluster-and-move.html to reassign ATLAS_ENTITIES/ATLAS_HOOK topic with new broker information. ... View more

@Owez Mujawar So you see the same exception when you try to publish a sample message to ATLAS_ENTITIES topic as atlas user. Some kafka config has been messed up - kafka brokers have been added or removed. Please follow https://community.hortonworks.com/content/supportkb/150063/how-to-add-new-kafka-brokers-to-a-cluster-and-move.html to reassign ATLAS_ENTITIES/ATLAS_HOOK topic with new broker information. ... View more

@Owez Mujawar Seems like you are missing jersey related jars in the impl directory again. Can you list all the jars in the impl directory? How was this cluster setup? The environment seems to miss many dependent jars ... View more

Looks like commons-configuration is not found in the classpath. Can you confirm if commons-configuration.jar is present in /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl directory. ... View more

@Owez Mujawar You see the NotificationException because Atlas is unable to send a message to kafka topic ATLAS_ENTITIES. Can you check if Kafka service is healthy? Another possible reason is when a new kafka broker was added and assigned leader and atlas doesn't know about it. To isolate this issue, can you try publishing a sample message to ATLAS_ENTITIES topic as atlas user and see if you see any permission issues. ... View more

can you please detail the steps you followed to install apache atlas ... View more

@Anji Raju looks like atlas server is not up and running, that might be the reason for service unavailable. Checking the log files could have helped, but you are not able to access the log files. Do you see log files created at all for atlas? ... View more

@Satya Nittala 1. Basic search uses POST command, please refer to this link for more details. 2. "excludeDeletedEntities": true flag ensures that the results will exclude deleted entities. 3. Can you elaborate, what didn't work if limit > 10000 ? ... View more

@Satya Nittala, you can use the following basic search API to return all hive_table entities in the cluster. You can change the value of limit and offset to get the result. curl -X POST http://172.27.17.5:21000/api/atlas/v2/search/basic \ -H 'Content-Type: application/json' \ -d '{ "excludeDeletedEntities": true, "includeSubClassifications": true, "includeSubTypes": true, "limit": 100000, "offset": 0, "typeName": "hive_table" }' ... View more

@Jeffrey Dwight Could you attach the full stack trace of the error? backend exception maybe result of many issues - solr/hbase down ... View more

curl -u <atlas_username>:<password> -X GET http://localhost:21000/api/atlas/v2/types/typedefs Can you try the above curl command - This will help to get the list of all typeDefs in atlas. You can follow the Atlas REST document for more details: http://atlas.apache.org/api/v2/index.html ... View more

@Megha U Is it mandatory to run the Ranger plugin on this external hive instance to get the Tagsync working? => No it is not mandatory to run ranger plugin on your external hive as long as you have the service with name = 'atlastest_hive' defined in ranger admin. 1. Can you confirm if you have the service with name defined 'atlastest_hive' in Ranger ? 2. If you have the service 'atlastest_hive' defined already, can you provide us with ranger admin logs to help debug further. ... View more

There are few issues with reusing the same entities after delete: 1. If the previously deleted entity has some tags - say PII associated with them, does the new table created inherit these tags? 2. If the new table is created with additional or lesser number of columns than the original, reusing tables will not help. If lineage cluttering is the issue, we can work towards adding filtering in lineage graph like exclude deleted entities from rendering. updating entities status from DELETED to ACTIVE doesn't help as only entity attribute updates are honored, but status, createTime, updateTime are all treated as system attributes of the entity. ... View more

I'm not sure if Cloudera's CDH provides support for Apache Atlas, maybe this question should be directed to the Cloudera community. ... View more

@Omar Abdeen The default propagation from hive_table to hive_column is set to NONE by default: { "name": "hive_table_columns", "typeVersion": "1.0", "relationshipCategory": "COMPOSITION", "endDef1": { "type": "hive_table", "name": "columns", "isContainer": true, "cardinality": "SET", "isLegacyAttribute": true }, "endDef2": { "type": "hive_column", "name": "table", "isContainer": false, "cardinality": "SINGLE", "isLegacyAttribute": true }, "propagateTags": "NONE" } You can update the relationshipDef - hive_table_columns to set "propagateTags": "ONE_TO_TWO" to enable propagation from table to column. ... View more

@Fernando Lopez Bello I think lineage is not broken when a table/view is created through Hive View version 1.0 and 2.0 (I just validated and it works). Could you check the atlas application.log and see if there any error stack trace relating to lineage. ... View more

@Rama Erigindindla, /usr/hdp/current/atlas-server symlinks to /usr/hdp/<hdp_version>/atlas e.g: /usr/hdp/2.6.2.0-56/atlas Can you check if hook-bin directory is present in /usr/hdp/<hdp_version>/atlas One possible cause could be the symlinks were not established right during upgrade. Please accept and upvote the answer if this helps. ... View more

@Laura Ngo, this is work in progress and we will be adding functionality to restrict certain tags from propagation using relationships. For more information on relationships: ATLAS-1690 ... View more

@Laura Ngo, lineage 'depth' is the number of hops on DataSet in each direction. The default lineage depth is 3 and default lineage direction is BOTH. To specify both lineage direction and depth you could use: curl -k -u admin:admin -H "Content-type:application/json" -X GET http://localhost:21000/api/atlas/v2/lineage/df8f62df-d956-4c82-828b-5e8ebc160d13?depth=5&direction=INPUT | python -m json.tool On the UI side, we don't pass the depth parameter as argument to the REST, hence it defaults to depth=3 in server-side. Currently we don't have a way to filter lineage entities based on tags or any filters, but this is a good use case and we will try to add this in future. ... View more

@M R we don't have out of box solution for Atlas-NiFi bridge. You can consider taking a look at the below solutions. https://community.hortonworks.com/repos/39432/nifi-atlas-lineage-reporter.html https://community.hortonworks.com/repos/66014/nifi-atlas-bridge.html Hope this helps. ... View more

@David Sheard, Atlas lineage are generated using native types DataSet and Process (e.g. DataSet -> Process -> DataSet). If you can represent your pipeline of loading data from HDFS into elastic search using Process and DataSet types, Alas will help you generate the lineage for the above pipeline (there is no out of box solution for the above transformation pipeline). You can view an example about creating lineage for custom types here. Hope this helps. If this answers your question please accept and up-vote the answer. ... View more

@Saurabh Currently the import-hive.sh script doesn't accept any restriction/option on import of databases or tables into Atlas. It uses hiveClient to get all databases and imports each database and tables associated with them. ... View more