About ssubramanian

ssubramanian · ‎12-21-2018

@Tim Mitchell can you confirm the contents of the following directory: [root@localhost atlas-hbase-plugin-impl]# ls -lt /usr/hdp/current/atlas-server/hook/hbase/atlas-hbase-plugin-impl total 17664 -rw-r--r--. 1 atlas hadoop 46464 Dec 6 04:32 hbase-bridge-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 13204 Dec 6 04:31 hdfs-model-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 19107 Dec 6 04:27 atlas-client-v2-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 61496 Dec 6 04:27 atlas-notification-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 7322236 Dec 6 04:27 kafka_2.11-2.0.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 1894753 Dec 6 04:27 kafka-clients-2.0.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 37544 Dec 6 04:27 atlas-client-common-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 69562 Dec 6 04:27 atlas-common-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 465346 Dec 6 04:26 atlas-intg-1.1.0.3.1.0.0-78.jar -rw-r--r--. 1 atlas hadoop 45927 Nov 13 10:54 jsr311-api-1.1.jar -rw-r--r--. 1 atlas hadoop 5749423 Nov 13 10:49 scala-library-2.11.12.jar -rw-r--r--. 1 atlas hadoop 66981 Nov 13 10:25 jackson-annotations-2.9.6.jar -rw-r--r--. 1 atlas hadoop 323848 Nov 13 10:25 jackson-core-2.9.6.jar -rw-r--r--. 1 atlas hadoop 1349339 Nov 13 10:25 jackson-databind-2.9.6.jar -rw-r--r--. 1 atlas hadoop 53275 Oct 22 2016 jersey-multipart-1.19.jar -rw-r--r--. 1 atlas hadoop 165345 Oct 21 2016 jersey-json-1.19.jar -rw-r--r--. 1 atlas hadoop 362679 Oct 21 2016 commons-configuration-1.10.jar You should have the listed jars for HBase hook to work correctly.

ssubramanian · ‎12-13-2018

@Karthikeyan Arjunan, sqoop hook is the way to import RDBMS data into Atlas. Currently we don't have hooks specific to RDBMS db like Oracle or Teradata

ssubramanian · ‎12-11-2018

currently there is no way to disable lineage rendering in UI for any table.

ssubramanian · ‎12-06-2018

Relationship category determines the style of relationship around containment and lifecycle. UML terminology is used for the values. ASSOCIATION is a relationship with no containment. COMPOSITION and AGGREGATION are containment relationships. The difference being in the lifecycles of the container and its children. In the COMPOSITION case, the children cannot exist without the container. For AGGREGATION, the life cycles of the container and children are totally independent.

ssubramanian · ‎12-03-2018

@Owez Mujawar you should have the following libraries for import-hive.sh to work. Not sure why you have lot of libraries. Did you manually copy atlas lib jars here?/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl [root@localhost atlas-hive-plugin-impl]# ls -ltr total 37448 -rw-r--r--. 1 atlas hadoop 4573750 May 11 2018 scala-reflect-2.11.8.jar -rw-r--r--. 1 atlas hadoop 802818 May 11 2018 scalap-2.11.8.jar -rw-r--r--. 1 atlas hadoop 5744974 May 11 2018 scala-library-2.11.8.jar -rw-r--r--. 1 atlas hadoop 15487351 May 11 2018 scala-compiler-2.11.8.jar -rw-r--r--. 1 atlas hadoop 1590487 May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 7532355 May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 67829 May 11 2018 json4s-native_2.11-3.2.11.jar -rw-r--r--. 1 atlas hadoop 589462 May 11 2018 json4s-core_2.11-3.2.11.jar -rw-r--r--. 1 atlas hadoop 82421 May 11 2018 json4s-ast_2.11-3.2.11.jar -rw-r--r--. 1 atlas hadoop 53275 May 11 2018 jersey-multipart-1.19.jar -rw-r--r--. 1 atlas hadoop 70540 May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 11639 May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 577458 May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 232482 May 11 2018 gson-2.5.jar -rw-r--r--. 1 atlas hadoop 408746 May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 98221 May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 222090 May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 64778 May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 18040 May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 43414 May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar -rw-r--r--. 1 atlas hadoop 35271 May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar

ssubramanian · ‎11-29-2018

@Owez Mujawar So you see the same exception when you try to publish a sample message to ATLAS_ENTITIES topic as atlas user. Some kafka config has been messed up - kafka brokers have been added or removed. Please follow https://community.hortonworks.com/content/supportkb/150063/how-to-add-new-kafka-brokers-to-a-cluster-and-move.html to reassign ATLAS_ENTITIES/ATLAS_HOOK topic with new broker information.

ssubramanian · ‎11-29-2018

@Owez Mujawar So you see the same exception when you try to publish a sample message to ATLAS_ENTITIES topic as atlas user. Some kafka config has been messed up - kafka brokers have been added or removed. Please follow https://community.hortonworks.com/content/supportkb/150063/how-to-add-new-kafka-brokers-to-a-cluster-and-move.html to reassign ATLAS_ENTITIES/ATLAS_HOOK topic with new broker information.

ssubramanian · ‎11-29-2018

@Owez Mujawar Seems like you are missing jersey related jars in the impl directory again. Can you list all the jars in the impl directory? How was this cluster setup? The environment seems to miss many dependent jars

ssubramanian · ‎11-28-2018

Looks like commons-configuration is not found in the classpath. Can you confirm if commons-configuration.jar is present in /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl directory.

ssubramanian · ‎11-28-2018

@Owez Mujawar You see the NotificationException because Atlas is unable to send a message to kafka topic ATLAS_ENTITIES. Can you check if Kafka service is healthy? Another possible reason is when a new kafka broker was added and assigned leader and atlas doesn't know about it. To isolate this issue, can you try publishing a sample message to ATLAS_ENTITIES topic as atlas user and see if you see any permission issues.

ssubramanian · ‎11-15-2018

can you please detail the steps you followed to install apache atlas

ssubramanian · ‎11-14-2018

@haosheng lou, consider creating a process entity with inputs and outputs attributes. e.g. POST http://localhost:21000/api/atlas/v2/entity { "entity": { "typeName": "Process", "attributes": { "description": null, "name": "process1", "owner": null, "inputs": [ { "guid": "cf3be027-1cbf-4109-a6b4-dabe980043a0", "typeName": "DataSet" } ], "outputs": [ { "guid": "8493a6a6-51c8-4106-a3dc-89eccb00833d", "typeName": "DataSet" } ], "qualifiedName": "process1@cl1" }, "guid": -1 }, "referredEntities": {} }

ssubramanian · ‎11-14-2018

@Anji Raju looks like atlas server is not up and running, that might be the reason for service unavailable. Checking the log files could have helped, but you are not able to access the log files. Do you see log files created at all for atlas?

ssubramanian · ‎10-26-2018

@Satya Nittala 1. Basic search uses POST command, please refer to this link for more details. 2. "excludeDeletedEntities": true flag ensures that the results will exclude deleted entities. 3. Can you elaborate, what didn't work if limit > 10000 ?

ssubramanian · ‎10-25-2018

@Satya Nittala, you can use the following basic search API to return all hive_table entities in the cluster. You can change the value of limit and offset to get the result. curl -X POST http://172.27.17.5:21000/api/atlas/v2/search/basic \ -H 'Content-Type: application/json' \ -d '{ "excludeDeletedEntities": true, "includeSubClassifications": true, "includeSubTypes": true, "limit": 100000, "offset": 0, "typeName": "hive_table" }'

ssubramanian · ‎10-14-2018

@Jeffrey Dwight Could you attach the full stack trace of the error? backend exception maybe result of many issues - solr/hbase down

ssubramanian · ‎09-28-2018

curl -u <atlas_username>:<password> -X GET http://localhost:21000/api/atlas/v2/types/typedefs Can you try the above curl command - This will help to get the list of all typeDefs in atlas. You can follow the Atlas REST document for more details: http://atlas.apache.org/api/v2/index.html

ssubramanian · ‎09-28-2018

@Megha U Is it mandatory to run the Ranger plugin on this external hive instance to get the Tagsync working? => No it is not mandatory to run ranger plugin on your external hive as long as you have the service with name = 'atlastest_hive' defined in ranger admin. 1. Can you confirm if you have the service with name defined 'atlastest_hive' in Ranger ? 2. If you have the service 'atlastest_hive' defined already, can you provide us with ranger admin logs to help debug further.

ssubramanian · ‎09-20-2018

@Gray Pickney Filtering in lineage graph is not present currently. There has been a lot of ask about this feature being included. We will work towards including this in our next atlas release.

ssubramanian · ‎09-19-2018

There are few issues with reusing the same entities after delete: 1. If the previously deleted entity has some tags - say PII associated with them, does the new table created inherit these tags? 2. If the new table is created with additional or lesser number of columns than the original, reusing tables will not help. If lineage cluttering is the issue, we can work towards adding filtering in lineage graph like exclude deleted entities from rendering. updating entities status from DELETED to ACTIVE doesn't help as only entity attribute updates are honored, but status, createTime, updateTime are all treated as system attributes of the entity.

ssubramanian · ‎09-19-2018

I'm not sure if Cloudera's CDH provides support for Apache Atlas, maybe this question should be directed to the Cloudera community.

ssubramanian · ‎09-19-2018

@Omar Abdeen The default propagation from hive_table to hive_column is set to NONE by default: { "name": "hive_table_columns", "typeVersion": "1.0", "relationshipCategory": "COMPOSITION", "endDef1": { "type": "hive_table", "name": "columns", "isContainer": true, "cardinality": "SET", "isLegacyAttribute": true }, "endDef2": { "type": "hive_column", "name": "table", "isContainer": false, "cardinality": "SINGLE", "isLegacyAttribute": true }, "propagateTags": "NONE" } You can update the relationshipDef - hive_table_columns to set "propagateTags": "ONE_TO_TWO" to enable propagation from table to column.

ssubramanian · ‎10-27-2017

@Fernando Lopez Bello I think lineage is not broken when a table/view is created through Hive View version 1.0 and 2.0 (I just validated and it works). Could you check the atlas application.log and see if there any error stack trace relating to lineage.

ssubramanian · ‎10-04-2017

HDP 3.0 will have Atlas 1.0.0, This is still under development and we will release a beta soon.

ssubramanian · ‎09-15-2017

@Rama Erigindindla, /usr/hdp/current/atlas-server symlinks to /usr/hdp/<hdp_version>/atlas e.g: /usr/hdp/2.6.2.0-56/atlas Can you check if hook-bin directory is present in /usr/hdp/<hdp_version>/atlas One possible cause could be the symlinks were not established right during upgrade. Please accept and upvote the answer if this helps.

ssubramanian · ‎09-13-2017

@Laura Ngo, this is work in progress and we will be adding functionality to restrict certain tags from propagation using relationships. For more information on relationships: ATLAS-1690

ssubramanian · ‎09-13-2017

The following GET works to list all type definitions in atlas: http://localhost:21000/api/atlas/v2/types/typedefs http://localhost:21000/api/atlas/v2/types/typedefs/headers Please refer to the following v2 REST API doc for reference: http://atlas.apache.org/api/v2/resource_TypesREST.html

ssubramanian · ‎09-11-2017

@Laura Ngo, lineage 'depth' is the number of hops on DataSet in each direction. The default lineage depth is 3 and default lineage direction is BOTH. To specify both lineage direction and depth you could use: curl -k -u admin:admin -H "Content-type:application/json" -X GET http://localhost:21000/api/atlas/v2/lineage/df8f62df-d956-4c82-828b-5e8ebc160d13?depth=5&direction=INPUT | python -m json.tool On the UI side, we don't pass the depth parameter as argument to the REST, hence it defaults to depth=3 in server-side. Currently we don't have a way to filter lineage entities based on tags or any filters, but this is a good use case and we will try to add this in future.

ssubramanian · ‎08-30-2017

@M R we don't have out of box solution for Atlas-NiFi bridge. You can consider taking a look at the below solutions. https://community.hortonworks.com/repos/39432/nifi-atlas-lineage-reporter.html https://community.hortonworks.com/repos/66014/nifi-atlas-bridge.html Hope this helps.

ssubramanian · ‎08-30-2017

@David Sheard, Atlas lineage are generated using native types DataSet and Process (e.g. DataSet -> Process -> DataSet). If you can represent your pipeline of loading data from HDFS into elastic search using Process and DataSet types, Alas will help you generate the lineage for the above pipeline (there is no out of box solution for the above transformation pipeline). You can view an example about creating lineage for custom types here. Hope this helps. If this answers your question please accept and up-vote the answer.

Online	Offline
Last Visited	‎10-21-2019 01:47 AM

Date Registered	‎07-25-2016 07:27 PM
Last Visited	‎10-21-2019 01:47 AM
Total Messages Posted	61
Total Kudos Received	32

Re: How can we create lineageinfo between two ent...

Re: Apache Atlas update DELETED entity to ACTIVE?

Re: Atlas Rest API V2 Error

Re: Messaging API with Kafka : lineage

Re: Searchable Tag hierarchy

Re: Atlas HBASE Hook jars missing

Re: Apache Atlas update DELETED entity to ACTIVE?

Re: Apache Atlas update DELETED entity to ACTIVE?

Re: Difference between different Apache Atlas rela...

Re: Issue while running import-hive.sh script

Re: Unable to Create entity in Atlas

Re: Unable to Create entity in Atlas

Re: Issue while running import-hive.sh script

Re: Issue while running import-hive.sh script

Re: Unable to Create entity in Atlas

Re: 503 Error while trying to test the status of a...

Re: How can we create lineageinfo between two ent...

Re: 503 Error while trying to test the status of a...

Re: How to get the active hive tables count in Apa...

Re: How to get the active hive tables count in Apa...

Re: How do you debug atlas startup?

Re: Atlas API rest Error

Re: Atlas Tags not getting synced to Ranger

Re: Apache Atlas update DELETED entity to ACTIVE?

Re: Apache Atlas update DELETED entity to ACTIVE?

Re: Apache Atlas Support with Cloudera

Re: How can classification be propagated at column...

Re: Atlas lineage tracking issue

Re: status of Atlas - is it in GA stage or technic...

Re: How to import Hive metadata to atlas (HDP 2.6....

Re: Do we have tag carry forward in atlas hdp2.6.1...

Re: Atlas Rest API V2 Error

Re: Atlas lineage REST API and UI query on depth

Re: Atlas , NiFi and Confluent Kafka

Re: Atlas Elasticsearch Hook