logs.zipI am trying to install Apache Atlas on a normal EC2 Instance by following the steps mentioned on Apache Atlas website. I have attached the logs. ... View more

logs.zipI am trying to install Apache Atlas 1.0.0 on AWS EC2 Instance (Amazon-linux-2) with embedded HBase and Solr. Somehow, I have managed to access REST API's (only on the AWS EC2 instance not on Local) but weburl (http://localhost:21000) is not working at all. I have followed the installation steps written on the apache atlas website. For referenece i have attached the application.log file and atlas-application.properties. Please note that firewall has been disabled and i have allowed all traffic for inbound and outbound rules. Let me know what i am doing wrong. Thank you in Advance, Subash ... View more

Hi, I tried both bulk and single Rest Endpoints (v1 and v2) to delete entities in Atlas. Both of them works mostly for deleting 3-4 entities and For rest other entities i am getting this weird error. {"error":"Failed to notify for change UPDATE"} Please note that Atlas UI becomes almost unresponsive while running the commands to delete entities. delete_guid = requests.delete(hostname+ '/api/atlas/entities?guid='+str(guid['$id$']['id']), headers=headers,auth =(username,password)) get_json=requests.delete(hostname+'/api/atlas/v2/entities/guids?'+guid_list, headers=headers,auth =(username,password)) get_json_new=requests.delete(hostname+'/api/atlas/v2/entity/bulk?' +guid_list, headers=headers,auth =(username,password)) We are using Atlas 0.8.0 Thank you, Subash ... View more

In Atlas 0.6 and 0.7, This Rest API command used to work perfectly to fetch the list of all tables available in one Hive database. Http://localhost:21000/api/atlas/discovery/search/dsl?query=hive_table+where+db.name%3D%22default%22&limit=1000 Let's say 100 tables are available in "default", 200 tables are in "default_raw", In such scenario, the total count of tables returned from Atlas (0.6 and 0.7) are 100 but in Atlas 0.8 I am getting 300 tables.Ideally, i should get only 100 tables. Let me know if DSL search Endpoints have been changed ... View more

Hi, As per the docs , I am trying the rest endpoint to fetch all entities in my browser. Rest Endpoint is : http://localhost:21000/api/atlas/v2/entity/bulk I tried multiple combinations of filters, but till now haven't found any luck. Let me know what am I missing here. Please suggest a good doc which can be referred. Ty, Subash ... View more

Hi, I would like to connect to Hive via Knox using JDBC connection in python. Please note that my cluster is secured and SSL is enabled and we use LDAP to Login to Hive via Knox.I am able to establish the connection to Hive using DbViz and JDBC driver of Hive. I had gone through a lot of links and haven't found any good examples as well as recommendations to connect hive using python. Thank you in Advance, Subash ... View more

Hi, To capture real-time events occurring in Hive, I am thinking of writing a Hive Hook. As I am not well versed in Java, Can I use python to build Hive Hook? If I can't. How to implement or customize Atlas Hive Hook written in Java. https://github.com/apache/atlas/blob/master/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java Do Apache Airflow Hive Hook works in a similar fashion: https://pythonhosted.org/airflow/_modules/hive_hooks.html Thank you in Advance, Subash ... View more

Hi, I use to run this API command (Atlas 0.6) to fetch the schema details from the Atlas. My whole purpose was to identify the tagged and not tagged columns from the huge JSON output. In Atlas 0.6, For tags, One new key value pair use to get added for tagged columns, Key Name was '$traits$'.In Atlas 0.7, I am not getting any classifier for tagged and not tagged columns. Basically the Rest End point was mentioned below. atlas_hostname + '/api/atlas/discovery/search/dsl?query=hive_table+where+db.name%3D%22'+ database + '%22&limit=10000' Whole Code is here : atlas_hostname = 'hostname:21000' database = 'mydb' atlas_user_name = 'myuser' atlas_password = 'mypassword' headers = {'Accept': 'application/json, text/plain, */*','Content-Type': 'application/json; charset=UTF-8'} def atlas_get_request(atlas_hostname,database, atlas_user_name,atlas_password): try: list_of_table_json=requests.get(atlas_hostname + '/api/atlas/discovery/search/dsl?query=hive_table+where+db.name%3D%22'+ database + '%22&limit=10000' , headers=headers, auth=(atlas_user_name,atlas_password)) dict_of_tables= list_of_table_json.json() result=dict_of_tables['results'] return result except requests.exceptions.HTTPError as err: print (err) return ("invalid Credentials or UI is unresponsive") I use to invoke the function atlas get request to fetch the schema details result = atlas_get_request(atlas_hostname,database, atlas_user_name,atlas_password) From the huge JSON response saved in "result", i used to filter the column key value pairs, Tagged column used to have an extra Key Value pair. Extra Key was "$traits$" in Atlas 0.6. Now in the column array of each and every column (in Atlas 0.7) i am getting only 9 key value pairs. No key value pairs which has the Tag details. Keys of the JSON dict mentioned below (tagged and untagged Column). [u'comment', u'$id$', u'qualifiedName', u'description', u'$typeName$', u'owner', u'table', u'type', u'name'] I am using apache atlas 0.7. Let me know how to overcome this issue. Thank you in advance, Subash ... View more

Hi, I am calling a python script where i am passing a huge JSON as an input, Even though i have increased the length limit of attribute in ExecuteStreamCommand my JSON input is getting truncated. How to overcome this error. Thank you in advance, Subash ... View more

As i have seen that we can import and export bulk policies in ranger, Can we do same sort of operation to delete policies as well using Rest End Point. Let say if i have thousand of masking policies in Apache Ranger and i want to delete all of them. How can this task be achieved. Thank you in Advance, Subash ... View more

Recently, I have installed HDP 2.6.1 on 3 node cluster to test new features of Apache Atlas and Ranger. When i ran import-hive.sh I am getting this issue. Exception in thread "main" org.apache.atlas.hook.AtlasHookException: HiveMetaStoreBridge.main() failed. at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:650) Caused by: com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out Hbase, Kafka, Solr all services are running fine. After doing a lot of configuration changes in Atlas and other services, i haven't got any success.later on i tried request module of python. It seems that "get" requests are working well. "post" is giving issue. Few question I have: 1. I am able to create Tags and associate tag to Default database so ideally "admin" user has privilege to create delete and update entities. 2. Same should work for rest API, But why only "get" is working. Why not "post" Thank you, Subash ... View more

I have tried maybe 100 of times with some tweaks but every time i am facing the same error. u'Unable to deserialize json \\"{\\\\\\"structTypes\\\\\\": [], \\\\\\"traitTypes\\\\\\": [], \\\\\\"classTypes\\\\\\": [{\\\\\\"hierarchicalMetaTypeName\\\\\\": \\\\\\"org.apache.atlas.typesystem.types.ClassType\\\\\\", \\\\\\"superTypes\\\\\\": [], \\\\\\"attributeDefinitions\\\\\\": [{\\\\\\"name\\\\\\": \\\\\\"multiplicityRequired\\\\\\", \\\\\\"reverseAttributeName\\\\\\": \\\\\\"\\\\\\", \\\\\\"multiplicity\\\\\\": \\\\\\"required\\\\\\", \\\\\\"dataTypeName\\\\\\": \\\\\\"string\\\\\\", \\\\\\"isUnique\\\\\\": \\\\\\"true\\\\\\", \\\\\\"isIndexable\\\\\\": \\\\\\"true\\\\\\", \\\\\\"isComposite\\\\\\": \\\\\\"true\\\\\\"}], \\\\\\"typeDescription\\\\\\": \\\\\\"\\\\\\", \\\\\\"typeName\\\\\\": \\\\\\"test\\\\\\"}], \\\\\\"enumTypes\\\\\\": []}\\"'} I am following this post, here i guess post request is successful. Let me know where i am doing wrong. https://community.hortonworks.com/articles/124/atlas-api-tips-create-trait-type-example.html curl Command : curl -u atlas_user:password -H 'Content-Type: application/json; charset=UTF-8' -X POST -d '{"enumTypes": [],"structTypes": [],"traitTypes": [{"superTypes":[],"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.TraitType","typeName": "EXPIRES_ON", "attributeDefinitions":[{ "name": "expiry_date", "dataTypeName": "date", "multiplicity": "required","isComposite": false,"isUnique": false,"isIndexable": true,"reverseAttributeName": null}]}],"classTypes": []}' http://hostname:21000/api/atlas/types I am attaching my python code as well for reference. import requests import json hostname = 'hostname:21000' headers = {'Content-Type': 'application/json; charset=UTF-8'} atlas_user_name = 'username' atlas_password = 'password' create_type ={ "enumTypes": [], "structTypes": [], "traitTypes": [ { "superTypes":[], "hierarchicalMetaTypeName": "org.apache.atlas.typesystem.types.TraitType", "typeName": "EXPIRES_ON", "attributeDefinitions": [ { "name": "expiry_date", "dataTypeName": "date", "multiplicity": "required", "isComposite": "false", "isUnique": "false", "isIndexable": "true", "reverseAttributeName": "" } ] } ], "classTypes": [] } type_create_post = requests.post(hostname+ '/api/atlas/types' , headers=headers,data = json.dumps(create_type),auth =(atlas_user_name,atlas_password)) print type_create_post.json() entity_create_post = requests.post(hostname+ '/api/atlas/types' , headers=headers,data = json.dumps(create_entity),auth =(atlas_user_name,atlas_password)) print entity_create_post.json() ... View more