Member since
01-31-2016
96
Posts
92
Kudos Received
20
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1617 | 02-11-2019 01:04 PM | |
1713 | 12-06-2018 01:19 PM | |
1001 | 08-23-2018 06:22 AM | |
980 | 08-09-2018 11:29 AM | |
1237 | 03-29-2018 04:55 PM |
03-02-2018
12:04 PM
@Satya Nittala hive_table is a type and fields you mentioned like aliases ,columns,comment,createTime,db etc., are attributes of hive_table. Type can be updated using PUT (http://atlas.apache.org/api/v2/resource_TypesREST.html#resource_TypesREST_updateAtlasTypeDefs_PUT). This requires fetching the type definition and updating with new attribute. For example , Following GET REST call is used to fetch the hive_table type definition : http://atlashost:21000/api/atlas/v2/types/entitydef/name/hive_table
After fetching the type definition , new attribute definition can be added in the attributeDefs array as {
name: "new_attribute",
typeName: "string",
isOptional: true,
cardinality: "SINGLE",
valuesMinCount: 0,
valuesMaxCount: 1,
isUnique: false,
isIndexable: false
} name : name of the new attribute typename : data type of the attribute isOptional : if the entity can be created without providing values for the attribute. (Note : updating a type with new mandatory attribute is not allowed. While updating , provide isOptional as True). and the updated JSON can be PUT to http://atlashost:21000/api/atlas/v2/types/typedefs For example , in the text file attached , I have added new attribute definition . GUID of the hive_table has to be modified based on your Atlas instance. Please let me know if you are stuck somewhere in this procedure. One question : hive_table is a defined type in Atlas.It has all attributes which will be required for maintaining hive meta data. May I know why you want to update it ? What is the new attribute you want to add ? could you please explain the use case behind it ?
... View more
03-02-2018
08:28 AM
2 Kudos
@Satya Nittala I hope you have enabled Atlas Hive hook settings. If yes , all updates to the hive table are captured by Atlas. When column is added in Hive, you can find the newly created hive_column entity in Atlas.
... View more
03-01-2018
06:53 AM
@priyal patel This log shows up when we stop Atlas. Can you please check the logs when you start Atlas and UI doesn't show up ?
... View more
02-28-2018
02:20 PM
1 Kudo
@Anurag Mishra please check https://hortonworks.com/tutorial/tag-based-policies-with-apache-ranger-and-apache-atlas/ .
... View more
02-28-2018
02:17 PM
1 Kudo
@priyal patel Could you please check application logs for any exceptions and if there are any , can you please paste here ? also , please check network tab and console tab for any errors. What is the response of http://atlashost:21000 in console tab ? and what is displayed on UI ? Can you check if the issue is https://issues.apache.org/jira/browse/ATLAS-1199 ?
... View more
02-28-2018
10:52 AM
3 Kudos
@Satya Nittala , hive_table entity can be edited on UI if the property is set in conf/application-properties file of Atlas To edit hive_table entities alone : atlas.ui.editable.entity.types=hive_table To edit entities of all types: atlas.ui.editable.entity.types=*
... View more
02-28-2018
10:00 AM
3 Kudos
@Satya Nittala Entity attributes can be updated using Entity Rest APIs (http://atlas.apache.org/api/v2/resource_EntityREST.html) Simple way to update an attribute using PUT using curl : <GUID of table> is the GUID of the hive_table entity : curl -v -u admin:admin -H "Content-Type: application/json" -X PUT -d '"table_description"' "http://localhost:21000/api/atlas/v2/entity/guid/<GUID of table>?name=description"
... View more
02-19-2018
01:40 PM
There is no such V2 REST API currently. For this requirement , you can use the search APIs. Example : /api/atlas/v2/search/basic?typeName=type_name /api/atlas/v2/search/dsl?typeName=type_name
... View more
02-19-2018
01:31 PM
@Laura Ngo Once you have the super tag "security_protection"created , following can be POSTed to API to create Types REST v2 API sub classifications. Note that the superTypes list contains the security_protection classification. http://localhost:21000/api/atlas/v2/types/typedefs?type=classification {
"classificationDefs":[
{
"name":"disk_encryption",
"description":"",
"superTypes":["security_protection" ],
"attributeDefs":[]
},
{
"name":"field_encryption",
"description":"",
"superTypes":["security_protection" ],
"attributeDefs":[]
}
]
}
... View more
02-18-2018
06:34 PM
@subash sharma To use the /api/atlas/v2/entity/bulk to fetch entities , GUIDs need to be known before and to be given as parameters. http://localhost:21000/api/atlas/v2/entity/bulk?guid=<guid1>&guid=guid2&guid=guid3
>> "fetch all entities in my browser" - There is no API currently to fetch all entities without specifying type / query / classification. Could you please explain your requirement ? Do the entities you require are of a specific type , or have a specific tag or perhaps have a pattern in name ?
... View more
11-22-2017
04:24 AM
1 Kudo
@Benjamin Hopp Following queries work : hive_table where db.name="db_name" hive_table where db.qualifiedName="db_qualifiedname" screen-shot-2017-11-22-at-95121-am.png
... View more
10-25-2017
06:55 AM
1 Kudo
@Aaron Mayo
GUIDS are generated by Atlas.Each Atlas entity has a unique GUID. You can get GUID of a table from UI , or by firing search query and writing a script on top of it to parse the GUIDS in the resulting JSON. For example , if you want to fetch the GUIDs of all tables in database default , DSL query = hive_db where db.name="default" Encoded : http://atlas_host:21000/api/atlas/v2/search/dsl?offset=0&query=db.name%3D%22default%22&typeName=hive_table Following is the JSON response attached as an image : dsl-query-result.png From this , json["entities"][0]["guid"],json["entities"][1]["guid"] ... json["entities"][n]["guid"] can be extracted. If you want to get GUID of only 1 table , instead of writing script , you can get it directly from the UI. When you click on an hive_table entity in Atlas, it takes you to http://atlas_host:21000/#!/detailPage/<GUID of the entity> Example : http://atlas_host:21000/#!/detailPage/c44d0207-3567-4573-baaf-577ecbb8e195 Here c44d0207-3567-4573-baaf-577ecbb8e195 is the GUID of the hive_table entity . Since , going to UI and getting the GUID for all tables is tedious and manual process , it is preferred to fire query which fetches required tables and fetch GUIDS using a script.
... View more
10-24-2017
06:07 PM
2 Kudos
@Aaron Mayo
This can be accomplished using V2 APIs.
POST request body example :
{ "classification":{ "typeName":"PII","attributes":{ "attrib1":"value1","attrib2":"value2"}},"entityGuids":[ "05c97069-dc36-4f26-b017-13582c42428a","4b3fb1fa-0755-4329-8ecb-7e53e18ed128","6b3e5b42-e09b-428a-89ad-9ae38690044a","7593f4ed-9cd2-46ba-b0a7-b92229301476","f21ee4aa-461c-4eeb-8945-31ac8ec648d6"]} API: http://atlas_host:21000/api/atlas/v2/entity/bulk/classification In the JSON , i) PII is the tag name ii)attributes is the attribute name , value map iii)"entityGuids" json array is the list of the GUIDs of the entities that have to be tagged. Please refer http://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_addClassification_POST
... View more
08-07-2017
10:33 AM
1 Kudo
@Dinesh Das , you must have specified tcp://10.74.58.106:61616 for "messaging" endpoint which might be incorrect .The value to be specified is located in "broker.url" in Falcon Advanced Configs.(Falcon startup properties)
... View more
08-04-2017
12:13 PM
1 Kudo
@Dinesh Das Check the interface endpoint specified for type = "messaging". (i.e) tcp://10.74.58.106:61616 is incorrect.
... View more
07-28-2017
07:03 AM
4 Kudos
@Paul Hanke 1.Add the following property in ATLAS_HOME/conf/atlas-application.properties : atlas.ui.editable.entity.types=your_custom_type or atlas.ui.editable.entity.types=* ( to list all types) 2. Restart Atlas. 3.Refresh browser cache Now you should be able to see the custom type you created.
... View more
07-11-2017
12:06 PM
1 Kudo
@Saba Baig Please refer to this issue https://issues.apache.org/jira/browse/ATLAS-1788
... View more
07-11-2017
12:04 PM
1 Kudo
2.1) What is a classification name? 2.2) Difference between tag and classification name? Classification and tags are same Basic search is basically Lucene search . So ,If you see your result in the screenshot, basic search has returned everything related to fact.(i.e) entities with description containing fact.
... View more
07-11-2017
10:39 AM
1 Kudo
@Saba Baig , >> But when I tried to the same search (Basic using query) I typed sales_fa?t and I got invalid Expression What is the query you fired ? You mentioned http://localhost:21000/api/atlas/v2/search/basic?limit=25∈cludeDeletedEntities=true&query=sales_fa?t&typeName=Table works for you. I am not able to get your question.
... View more
07-11-2017
08:30 AM
1 Kudo
@Saba Baig >> Also, what features e.g wildcards (? . * etc) are available in full-text provided by Atlas? and how can we use them (examples)? Atlas uses lucene query syntax for its full text search (basic search). Please refer lucene query usage here : http://lucene.apache.org/core/4_1_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#Wildcard_Searches Example: If I have 3 hive_table entities in Atlas , table_a , table_b , table_ab Query :
1)
type = hive_table , query = table_?
http://localhost:21000/api/atlas/v2/search/basic?limit=25&excludeDeletedEntities=true&query=table_?&typeName=hive_table
Returns 2 entities (table_a , table_b)
2)
type = hive_table , query = table_*
http://localhost:21000/api/atlas/v2/search/basic?limit=25&excludeDeletedEntities=true&query=table_*&typeName=hive_table
Returns 3 entities (table_a , table_b , table_ab) I hope this example makes it clear ... Note : Wild card search can be done on UI itself.
... View more
07-11-2017
08:20 AM
1 Kudo
@Saba BaigFew aggregate functions are not working in V2 APIs. There are few Apache JIRAs filed for the same : https://issues.apache.org/jira/browse/ATLAS-1874 https://issues.apache.org/jira/browse/ATLAS-1639 However V1 APIs work perfectly . For example :
Query : hive_table groupby(owner) select owner,count()
URL encoded :
http://localhost:21000/api/atlas/discovery/search/dsl?limit=25&query=hive_table+groupby(owner)+select+owner%2Ccount()
Response :
{requestId: "pool-2-thread-9 - 8a45b7a6-0cd9-4e9d-8b71-66e313f5abf1",
query: "hive_table groupby(owner) select owner,count()",
queryType: "dsl",
count: 2,
results:
[
{$typeName$: "__tempQueryResultStruct74",
count(): 9,
owner: "hrt_qa"
},
{$typeName$: "__tempQueryResultStruct74",
count(): 2,
owner: "anonymous"
}
],
dataType:
{typeName: "__tempQueryResultStruct74",
typeDescription: null,
typeVersion: "1.0",
attributeDefinitions:
[
{name: "owner",
dataTypeName: "string",
multiplicity:
{lower: 0,
upper: 1,
isUnique: false
},
isComposite: false,
isUnique: false,
isIndexable: false,
reverseAttributeName: null
},
{name: "count()",
dataTypeName: "long",
multiplicity:
{lower: 0,
upper: 1,
isUnique: false
},
isComposite: false,
isUnique: false,
isIndexable: false,
reverseAttributeName: null
}
]
}
}
This query groups the owner of hive_table and and lists the owner ,count in each group . The concept is similar to that of SQL. Since the UI currently uses V2 query , you may not see the results on UI. But you can always rely on REST APIs. For this particular kind of queries , please use V1 APIs for now .
... View more
07-10-2017
01:05 PM
1 Kudo
@Saba Baig The select lets you "select" a particular attribute of an entity DB where name="Reporting" select name, owner displays the name and owner of db whose name is "Reporting" Currently UI doesn't display results but REST GET API call actually returns the result. Example : DSL query : hive_table name = "table1" select qualifiedName,owner response : {queryType: "DSL",
queryText: "`hive_table` name="table1" select qualifiedName,owner",
attributes:
{name:
["qualifiedName",
"owner"
],
values:
[
["default.table1@cl1",
"atlas"
]
]
}
}
... View more
06-29-2017
10:40 AM
3 Kudos
@subash sharma Try this : 1. Changed "" to None 2.Changed String false , true to boolean True ,False 3. Added typeDescription and typeVersion create_type = {
"enumTypes": [],
"structTypes": [],
"traitTypes": [
{
"superTypes": [],
"hierarchicalMetaTypeName":
"org.apache.atlas.typesystem.types.TraitType",
"typeName": "EXPIRES_ON",
"typeDescription": None,
"typeVersion": "1.0",
"attributeDefinitions": [
{
"name": "expiry_date",
"dataTypeName": "date",
"multiplicity": "required",
"isComposite": False,
"isUnique":False,
"isIndexable": True,
"reverseAttributeName": None}
]
}
],
"classTypes": []
}
... View more
06-23-2017
04:00 PM
1 Kudo
@Smart Data Can you please try with "--broker-list localhost:6667" ? broker seems to be running on port 6667. To verify the port number on which kafka broker is running , get into zookeeper client shell using $ZOOKEEPER_HOME/bin/zkCli.sh and get the broker port. The following image is taken after running zookeeper client shell and note that get /brokers/ids/0 lists the port. Best way to check the processes running on a port is using lsof -i:6667. In your case , "kakfa" in the 3rd column of output lsof -i -P -n | grep kafka is the kafka user and not the process itself. Also , best practice is to use the hostname itself instead of "localhost".
... View more
06-23-2017
02:27 PM
3 Kudos
@Smart Data , Can you please check if the port 9092 is correct in the broker list and the broker is up and running ? Also , is your cluster kerberized ? Please note that : The issue has nothing to do with Atlas or Ranger . When Atlas starts up , it creates 2 kafka topics ATLAS_HOOK and ATLAS_ENTITIES . The user doesn't have to create any topic.
... View more
06-15-2017
08:41 AM
3 Kudos
@Saba Baig >> How Hive notifies Atlas about any DML/DDL operation in Atlas against which Atlas generates lineage? Whenever there is any metadata change events in Hive , HiveHook captures it and puts the details of created/updated hive entity to a kafka topic called ATLAS_HOOK. Atlas is the consumer of the ATLAS_HOOK. So Atlas gets the message from ATLAS_HOOK. >> what is the information that Hive sends to Atlas? Example : hive > create table emp(id int,name string); 1.HiveHook composes a JSON message that contains information about table name , database , columns and other table properties and sends it to ATLAS_HOOK. 2. ATLAS_HOOK queues up the messages from HiveHook and Atlas consumes from it. Atlas consumes the JSON message about table emp and ingests it. hive > create table t_emp as select * from emp; 1.HiveHook composes JSON message that contains t_emp details and also the source table name (emp) and sends to ATLAS_HOOK. 2.Atlas understands from the JSON message consumed from ATLAS_HOOK , that it is a CTAS table and it has a source table , ingests the table t_emp and constructs lineage for the tables emp and t_emp. >> is Hive DB going to notify Atlas Server on its own or HiveHook is going to check constantly in the Hive DB and pull the changes HiveHook doesn't check hive constantly all time. Whenever there is any metadata event change ( like when user fires a hive query that involves creation/updation/drop ) , HiveHook notifies ATLAS_HOOK. NOTE : If you want to know more about the exact JSON content sent by HiveHook , you can create a table in hive and check the message that lands in ATLAS_HOOK for that table.
... View more
06-14-2017
09:17 AM
1 Kudo
@subash sharma , DSL query makes GREMLIN call internally to fetch results.
... View more
06-12-2017
10:01 AM
1 Kudo
@subash sharma , limit parameter should work . How many tables are there in the database you are querying?
... View more
06-12-2017
09:29 AM
2 Kudos
@subash sharma , add limit parameter like : /api/atlas/discovery/search/dsl?query=hive_table+where+db.name%3D%22default%22&limit=500
... View more
06-12-2017
08:43 AM
1 Kudo
Welcome @subash sharma . Please accept the answer if it answered your question.
... View more
- « Previous
-
- 1
- 2
- Next »