Created 07-11-2017 05:46 PM
Hello
I'm able to create a new entity for the type hive_table using V1 rest api. But I'm not that clear on how to do the same using V2 API, in particular when some attributes (columns) of hive_table are referencing other types (hive_column).
Any help would be appreciated.
Thanks
Carlos
,Hello
I'm trying to use Atlas V2 rest apis to create an entity for the hive_table type. I'm able to do that via V1, but it's not clear to me on how to do this via V2, in particular when some attributes (columns) of hive_table are pointing to another type (hive_column).
Any help would be appreciated
Thanks!
Carlos
Created 07-11-2017 06:14 PM
Hope this helps.
curl -X POST \ http://localhost:21000/api/atlas/v2/entity \ -H 'authorization: Basic YWRtaW46YWRtaW4=' \ -H 'content-type: application/json' \ -d '{ "entity": { "guid": "-208942807557404", "status": "ACTIVE", "version": 0, "typeName": "hive_table", "attributes": { "owner": "hive", "temporary": false, "name": "test_table_0001", "qualifiedName": "my_db.test_table_0001@hdp_cluster", "columns": [ { "guid": "-208942807557402", "typeName": "hive_column" }, { "guid": "-208942807557401", "typeName": "hive_column" }, { "guid": "-208942807557400", "typeName": "hive_column" } ], "tableType": "MANAGED_TABLE", "sd": { "guid": "-208942807557403", "typeName": "hive_storagedesc" }, "partitionKeys": [ { "guid": "-208942807557405", "typeName": "hive_column" } ], "parameters": { "totalSize": "0", "rawDataSize": "0", "numRows": "0", "COLUMN_STATS_ACCURATE": "{\"BASIC_STATS\":\"true\"}", "numFiles": "0", "transient_lastDdlTime": "1497764442" }, "db": { "guid": "90a7d3af-873a-4c10-a815-069f2d47d490", "typeName": "hive_db" }, "retention": 0 } } , "referredEntities": { "-208942807557403": { "guid": "-208942807557403", "status": "ACTIVE", "version": 0, "typeName": "hive_storagedesc", "attributes": { "qualifiedName": "my_db.test_table_0001@hdp_cluster_storage", "storedAsSubDirectories": false, "location": "hdfs://localhost.localdomain:8020/apps/hive/warehouse/my_db.db/test_table_0001", "compressed": false, "inputFormat": "org.apache.hadoop.mapred.TextInputFormat", "outputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", "parameters": {}, "serdeInfo": { "typeName": "hive_serde", "attributes": { "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe", "parameters": { "serialization.format": "1" } } }, "table": { "guid": "-208942807557404", "typeName": "hive_table" }, "numBuckets": -1 } }, "-208942807557402": { "guid": "-208942807557402", "status": "ACTIVE", "version": 0, "typeName": "hive_column", "attributes": { "owner": "hive", "qualifiedName": "my_db.test_table_0001.col1@hdp_cluster", "name": "col1", "position": 0, "type": "int", "table": { "guid": "-208942807557404", "typeName": "hive_table" } } }, "-208942807557401": { "guid": "-208942807557401", "status": "ACTIVE", "version": 0, "typeName": "hive_column", "attributes": { "owner": "hive", "qualifiedName": "my_db.test_table_0001.col2@hdp_cluster", "name": "col2", "position": 1, "type": "string", "table": { "guid": "-208942807557404", "typeName": "hive_table" } } }, "-208942807557400": { "guid": "-208942807557400", "status": "ACTIVE", "version": 0, "typeName": "hive_column", "attributes": { "owner": "hive", "qualifiedName": "my_db.test_table_0001.col3@hdp_cluster", "name": "col3", "position": 2, "type": "string", "table": { "guid": "-208942807557404", "typeName": "hive_table" } } }, "-208942807557405": { "guid": "-208942807557405", "status": "ACTIVE", "version": 0, "typeName": "hive_column", "attributes": { "owner": "hive", "qualifiedName": "my_db.test_table_0001.col4@hdp_cluster", "name": "col4", "position": 5, "type": "string", "table": { "guid": "-208942807557404", "typeName": "hive_table" } } } } }'
Created 07-11-2017 06:14 PM
V2 REST documentation can be found here
atlas.incubator.apache.org/api/v2/index.html
If you find the answer useful, please accept and upvote.
Created 07-21-2017 04:58 PM
Hello @Sarath Subramanian
Thank you for your info. One question, it seems that 'referredEntities' is not needed as long as those are embedded within the 'entity' attribute, correct?
,Hello @Sarath Subramanian
Thank you for your info. One more question, it seems that we don't need 'referredEntities' attribute in the request as long as those entities are already present within the 'entity' attribute, correct?
Created 07-21-2017 05:40 PM
@Carlos Guerra, 'entity' attributes should contain only object id references. More information about the referred object id entity should be added in the 'referredEntities' section.
For e.g. when creating hive_table entity, hive_column attributes are referred using its object ids in 'entity' section and detailed information about the hive column (like its own attributes) are defined in the referred entities section.
If this answers your question please accept and upvote.
Created 07-22-2017 04:26 PM
Hello @Sarath Subramanian
Thanks for your response. One other question, I just saw that you made a commit on
ATLAS-1807 : Enhance DSL query to support like operator for wildcard
Could you please show a DSL query example of your 'like' implementation? I'd like to use it. I did a build on May 26th and I must have your commit.
Thanks
Carlos
Created 07-24-2017 06:37 AM
@Carlos Guerra, some sample dsl like queries:
Created 07-24-2017 04:17 PM
hello @Sarath Subramanian
that's great! thank you 😉
Created 07-25-2017 06:27 PM
sorry to bother you. 'like' seems to work with single types. but for arrays it doesn't seem to work. for example, i have an entity my_entity with a property my_attributes of typeName array<my_attribute> where my_attribute is of structDefs
{ "name": "my_attribute",
"typeVersion": "1.0",
"attributeDefs": [
{ "name": "name",
"typeName": "string",
"cardinality": "SINGLE",
"isIndexable": true,
"isOptional": false,
"isUnique": false },
{ "name": "value",
"typeName": "string",
"cardinality": "SINGLE",
"isIndexable": true,
"isOptional": false,
"isUnique": false } ] }
if i do a DSL query: my_entity where my_attributes.name like "*lang*" i get an exception like this:
2017-07-25 14:05:48,225 ERROR - [pool-1-thread-4 - 05457522-89ab-4f1b-a8ba-3134413d1838:] ~ Error handling a request: 5d7cba60f2da2a76 (ExceptionMapperUtil:32) org.apache.atlas.exception.AtlasBaseException: Gremlin script execution failed: L:{def r=(([]) as Set);def f1={GremlinPipeline x->x.as('a0').out('__my_entity.my_attributes').as('__res') [0..<25].select(['a0', '__res']).fill(r)};f1(g.V().has('__typeName','my_entity'));f1(g.V().has('__superTypeNames','my_entity'));r._().as('__tmp').transform({((Row)it).getColumn('a0')}).as('a0').back('__tmp').transform({((Row)it).getColumn('__res')}).as('__res').filter({it.'my_attribute.name'.matches('*lang*')}).back('a0') [0..<25].toList()} at org.apache.atlas.repository.graphdb.titan0.Titan0Graph.executeGremlinScript(Titan0Graph.java:342) at org.apache.atlas.repository.graphdb.titan0.Titan0Graph.executeGremlinScript(Titan0Graph.java:274) at org.apache.atlas.discovery.EntityDiscoveryService.searchUsingDslQuery(EntityDiscoveryService.java:114) at org.apache.atlas.web.rest.DiscoveryREST.searchUsingDSL(DiscoveryREST.java:97)
do you know what i'm doing wrong here? or is this not yet supported?
thanks
carlos
Created 07-25-2017 07:51 PM
i'm also getting this issue with attributes that are referring other entities. not just arrays. i have a built from may 2017. fyi
carlos