Atlas V2 create entity API,atlas v2 api examples

[#1]

Hello

I'm able to create a new entity for the type hive_table using V1 rest api. But I'm not that clear on how to do the same using V2 API, in particular when some attributes (columns) of hive_table are referencing other types (hive_column).

Any help would be appreciated.

Thanks

Carlos

,

Hello

I'm trying to use Atlas V2 rest apis to create an entity for the hive_table type. I'm able to do that via V1, but it's not clear to me on how to do this via V2, in particular when some attributes (columns) of hive_table are pointing to another type (hive_column).

Any help would be appreciated

Thanks!

Carlos

[#2]

@Carlos Guerra, you can use the sample curl script to create hive_table entity using v2 APIs. Please note that all the referred entities like hive_column, hive_storagedesc are referred using negative long guids, also a valid db guid is specified for the db attribute.

Hope this helps.

curl -X POST \
  http://localhost:21000/api/atlas/v2/entity \
  -H 'authorization: Basic YWRtaW46YWRtaW4=' \
  -H 'content-type: application/json' \
  -d '{
  "entity":
    {
      "guid": "-208942807557404",
      "status": "ACTIVE",
      "version": 0,
      "typeName": "hive_table",
      "attributes": {
        "owner": "hive",
        "temporary": false,
        "name": "test_table_0001",
        "qualifiedName": "my_db.test_table_0001@hdp_cluster",
        "columns": [
          {
            "guid": "-208942807557402",
            "typeName": "hive_column"
          },
          {
            "guid": "-208942807557401",
            "typeName": "hive_column"
          },
          {
            "guid": "-208942807557400",
            "typeName": "hive_column"
          }
        ],
        "tableType": "MANAGED_TABLE",
        "sd": {
          "guid": "-208942807557403",
          "typeName": "hive_storagedesc"
        },
        "partitionKeys": [
          {
            "guid": "-208942807557405",
            "typeName": "hive_column"
          }
        ],
        "parameters": {
          "totalSize": "0",
          "rawDataSize": "0",
          "numRows": "0",
          "COLUMN_STATS_ACCURATE": "{\"BASIC_STATS\":\"true\"}",
          "numFiles": "0",
          "transient_lastDdlTime": "1497764442"
        },
        "db": {
          "guid": "90a7d3af-873a-4c10-a815-069f2d47d490",
          "typeName": "hive_db"
        },
        "retention": 0
      }
    }
  ,
  "referredEntities": {
    "-208942807557403": {
      "guid": "-208942807557403",
      "status": "ACTIVE",
      "version": 0,
      "typeName": "hive_storagedesc",
      "attributes": {
        "qualifiedName": "my_db.test_table_0001@hdp_cluster_storage",
        "storedAsSubDirectories": false,
        "location": "hdfs://localhost.localdomain:8020/apps/hive/warehouse/my_db.db/test_table_0001",
        "compressed": false,
        "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
        "outputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
        "parameters": {},
        "serdeInfo": {
          "typeName": "hive_serde",
          "attributes": {
            "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
            "parameters": {
              "serialization.format": "1"
            }
          }
        },
        "table": {
          "guid": "-208942807557404",
          "typeName": "hive_table"
        },
        "numBuckets": -1
      }
    },
    "-208942807557402": {
      "guid": "-208942807557402",
      "status": "ACTIVE",
      "version": 0,
      "typeName": "hive_column",
      "attributes": {
        "owner": "hive",
        "qualifiedName": "my_db.test_table_0001.col1@hdp_cluster",
        "name": "col1",
        "position": 0,
        "type": "int",
        "table": {
          "guid": "-208942807557404",
          "typeName": "hive_table"
        }
      }
    },
    "-208942807557401": {
      "guid": "-208942807557401",
      "status": "ACTIVE",
      "version": 0,
      "typeName": "hive_column",
      "attributes": {
        "owner": "hive",
        "qualifiedName": "my_db.test_table_0001.col2@hdp_cluster",
        "name": "col2",
        "position": 1,
        "type": "string",
        "table": {
          "guid": "-208942807557404",
          "typeName": "hive_table"
        }
      }
    },
    "-208942807557400": {
      "guid": "-208942807557400",
      "status": "ACTIVE",
      "version": 0,
      "typeName": "hive_column",
      "attributes": {
        "owner": "hive",
        "qualifiedName": "my_db.test_table_0001.col3@hdp_cluster",
        "name": "col3",
        "position": 2,
        "type": "string",
        "table": {
          "guid": "-208942807557404",
          "typeName": "hive_table"
        }
      }
    },
    "-208942807557405": {
      "guid": "-208942807557405",
      "status": "ACTIVE",
      "version": 0,
      "typeName": "hive_column",
      "attributes": {
        "owner": "hive",
        "qualifiedName": "my_db.test_table_0001.col4@hdp_cluster",
        "name": "col4",
        "position": 5,
        "type": "string",
        "table": {
          "guid": "-208942807557404",
          "typeName": "hive_table"
        }
      }
    }
  }
}'

[#3]

V2 REST documentation can be found here

atlas.incubator.apache.org/api/v2/index.html

If you find the answer useful, please accept and upvote.

[#4]

Hello @Sarath Subramanian

Thank you for your info. One question, it seems that 'referredEntities' is not needed as long as those are embedded within the 'entity' attribute, correct?

,

Hello @Sarath Subramanian

Thank you for your info. One more question, it seems that we don't need 'referredEntities' attribute in the request as long as those entities are already present within the 'entity' attribute, correct?

[#5]

@Carlos Guerra, 'entity' attributes should contain only object id references. More information about the referred object id entity should be added in the 'referredEntities' section.

For e.g. when creating hive_table entity, hive_column attributes are referred using its object ids in 'entity' section and detailed information about the hive column (like its own attributes) are defined in the referred entities section.

If this answers your question please accept and upvote.

[#6]

Hello @Sarath Subramanian

Thanks for your response. One other question, I just saw that you made a commit on

ATLAS-1807 : Enhance DSL query to support like operator for wildcard

Could you please show a DSL query example of your 'like' implementation? I'd like to use it. I did a build on May 26th and I must have your commit.

Thanks

Carlos

[#7]

@Carlos Guerra, some sample dsl like queries:

• hive_table where name like "sa?es*"
• hive_db where name like "R*"
• hive_db where hive_db.name like "R???rt?*" or hive_db.name like "S?l?s" or hive_db.name like "Log*"
• hive_db where hive_db.name like "R???rt?*" and hive_db.name like "S?l?s" and hive_db.name like "Log*"
• hive_table where name like 'sales*', db where name like 'Sa?es'

[#8]

hello @Sarath Subramanian

that's great! thank you 😉

[#9]

hi @Sarath Subramanian

sorry to bother you. 'like' seems to work with single types. but for arrays it doesn't seem to work. for example, i have an entity my_entity with a property my_attributes of typeName array<my_attribute> where my_attribute is of structDefs

{ "name": "my_attribute",

"typeVersion": "1.0",

"attributeDefs": [

{ "name": "name",

"typeName": "string",

"cardinality": "SINGLE",

"isIndexable": true,

"isOptional": false,

"isUnique": false },

{ "name": "value",

"typeName": "string",

"cardinality": "SINGLE",

"isIndexable": true,

"isOptional": false,

"isUnique": false } ] }

if i do a DSL query: my_entity where my_attributes.name like "*lang*" i get an exception like this:

2017-07-25 14:05:48,225 ERROR - [pool-1-thread-4 - 05457522-89ab-4f1b-a8ba-3134413d1838:] ~ Error handling a request: 5d7cba60f2da2a76 (ExceptionMapperUtil:32) org.apache.atlas.exception.AtlasBaseException: Gremlin script execution failed: L:{def r=(([]) as Set);def f1={GremlinPipeline x->x.as('a0').out('__my_entity.my_attributes').as('__res') [0..<25].select(['a0', '__res']).fill(r)};f1(g.V().has('__typeName','my_entity'));f1(g.V().has('__superTypeNames','my_entity'));r._().as('__tmp').transform({((Row)it).getColumn('a0')}).as('a0').back('__tmp').transform({((Row)it).getColumn('__res')}).as('__res').filter({it.'my_attribute.name'.matches('*lang*')}).back('a0') [0..<25].toList()} at org.apache.atlas.repository.graphdb.titan0.Titan0Graph.executeGremlinScript(Titan0Graph.java:342) at org.apache.atlas.repository.graphdb.titan0.Titan0Graph.executeGremlinScript(Titan0Graph.java:274) at org.apache.atlas.discovery.EntityDiscoveryService.searchUsingDslQuery(EntityDiscoveryService.java:114) at org.apache.atlas.web.rest.DiscoveryREST.searchUsingDSL(DiscoveryREST.java:97)

do you know what i'm doing wrong here? or is this not yet supported?

thanks

carlos

[#10]

hi @Sarath Subramanian

i'm also getting this issue with attributes that are referring other entities. not just arrays. i have a built from may 2017. fyi

carlos