Is there a way of masking timestamps using the masking policies in Ranger? Perhaps through a custom UDF referenced in the policy? The masking type of "Date: show only year" only works for dates, not timestamps. Looking at GitHub (as the Ranger documentation isn't complete) timestamps cannot be masked, only nullified - https://github.com/myui/hive-udf-backports/tree/master/src/main/java/org/apache/hadoop/hive/ql/udf/generic mask(value, upperChar, lowerChar, digitChar, otherChar, numberChar, dayValue, monthValue, yearValue) Supported types: TINYINT, SMALLINT, INT, BIGINT, STRING, VARCHAR, CHAR, DATE Reason: some systems store dates as timestamps, so date of birth (PII) could be stored as a timestamp and I need to mask it. ... View more

I can only return a maximum of 100 results when doing a DSL API search in Atlas. Is this by design or a bug? Even with a limit of 1000 only 100 items are returned curl -k -u admin:admin -H "Content-type:application/json" -X GET https://url:port/api/atlas/v2/search/dsl?query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20qualifiedName%20like%20%27prod_%2A_data_lake%2A%27%20select%20qualifiedName%2Cname%2C__guid%20limit%201000 | python -m json.tool > hive_column_prod_data_lake_limit.json No limit yet still 100 items are returned. There are a lot more than 100 items - when I do the same for hive_tables only 100 items are returned. curl -k -u admim:admin -H "Content-type:application/json" -X GET https://url:port/api/atlas/v2/earch/dsl?query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20qualifiedName%20like%20%27prod_%2A_data_lake%2A%27%20selct%20qualifiedName%2Cname%2C__guid | python -m json.tool > hive_column_prod_data_lake.json This is on a 2.6.3 HDP install. ... View more

Is it possible to reference more than one Atlas tag in one Ranger policy via the Policy Conditions? I can set-up allow or deny tag policies, but would like to reference a combination of tags in the Policy Conditions on one policy. Is this possible? Example Let's say I have these 3 tags: (tenancy_component with some attributes, tenancy_xxx and tenancy_yyy). { "classificationDefs":[ { "createdBy": "Laura", "name": "tenancy_component", "description": "tenancy_component", "attributeDefs": [ { "name":"landing", "typeName":"boolean", "isOptional":"true", "isUnique":"false", "isIndexable":"true", "cardinality":"SINGLE" }, { "name":"staging", "typeName":"boolean", "isOptional":"true", "isUnique":"false", "isIndexable":"true", "cardinality":"SINGLE" }, { "name":"data_lake", "typeName":"boolean", "isOptional":"true", "isUnique":"false", "isIndexable":"true", "cardinality":"SINGLE" }], "superTypes": [] } ] } { "classificationDefs":[ { "createdBy": "Laura", "name": "tenancy_xxx", "description": "tenancy_xxx", "attributeDefs": [ { }], "superTypes": [] }, { "createdBy": "Laura", "name": "tenancy_yyy", "description": "tenancy_yyy", "attributeDefs": [ { }], "superTypes": [] } ] } I want to provide access (ABAC) to a role such that it doesn't have access to landing unless it is in tenancy xxx, it has access to the data lake for tenancy xxx but not yyy. The role only have access to staging if it is part of tenancy_yyy. Database name Tags Access db1 tenancy_xxx, tenancy_component.landing=true Access db2 tenancy_xxx, tenancy_component.staging=true Deny db3 tenancy_xxx, tenancy_component.data_lake=true Access db5 tenancy_yyy, tenancy_component.landing=true Deny db6 tenancy_yyy, tenancy_component.staging=true Access db7 tenancy_yyy, tenancy_component.data_lake=true Deny db7 tenancy_component.data_lake=true Deny How many tag policies should I have and how would I do it? ... View more

I want to mask some data. I'm testing in the 2.6.3 sandbox I have created a tag: {"category": "CLASSIFICATION", "guid": "bb29dc29-11ba-4d92-8d8f-fdca8ae92ea4", "createdBy": "holger_gov", "updatedBy": "holger_gov", "createTime": 1518326442355, "updateTime": 1518326442355, "version": 1, "name": "test_pii_tag", "description": "test_pii_tag", "typeVersion": "1.0", "attributeDefs": [ {"name": "masking_type", "typeName": "string", "isOptional": true, "cardinality": "SINGLE", "valuesMinCount": 0, "valuesMaxCount": 1, "isUnique": false, "isIndexable": false }, {"name": "last_4", "typeName": "boolean", "isOptional": true, "cardinality": "SINGLE", "valuesMinCount": 0, "valuesMaxCount": 1, "isUnique": false, "isIndexable": false } ], "superTypes": [], } I have tagged 4 columns on foodmart.customer with test_pii_tag and set the following attributes: lname (attribute string masking_type = "hash") fname (attribute string masking_type = "nullify") address1 (attribute boolean last_4 = true ) birthdate (attribute string masking_type = "year") I created one Ranger tag policy and set the following deny setting for raj_ops: Mask: Hive hash if ( tagAttr.get('masking_type').equals("hash") ) { ctx.result = true; } Mask: Hive nullify if ( tagAttr.get('masking_type').equals("nullify") ) { ctx.result = true; } Mask: Hive Date: show only year if ( tagAttr.get('masking_type').equals("year") ) { ctx.result = true; } Mask: Hive Partial mask show last 4 if ( tagAttr.get('last_4').equals("true") ) { ctx.result = true; } -- I also tried the below with the same results if ( tagAttr.get('last_4') ) { ctx.result = true; } When I run SELECT * FROM customer LIMIT 100; I see the following: lname is hashed - as expected fname null - as expected address1 is hashed - not as expected birthdate yyyy-01-01 as expected What is wrong with my javascript expressions to cause address1 to be hashed instead of 'Partial mask show last 4'? ... View more

There's documentation for creating a child/parent relationship for classifications in the UI. What would be the equivalent script for creating the same relationship in the API? Let's say I have a parent classification named 'security_protection'. I want to create child classifications via API request named 'disk_encryption' and field_encryption'. What would be the v2 API request to do this? Thanks in advance. Version 2.6.3 ... View more

Please could someone provide me with an example AtlasEntity REST API POST query I can use on my sandbox please. I'm struggling to post a successful one with v0.8. I managed to create a type OK. Here's my entity type. POST http://127.0.0.1:21000/api/atlas/v2/types/typedefs { "enumDefs":[], "structDefs":[], "classificationDefs":[], "entityDefs":[ { "superTypes":[ "DataSet" ], "name":"test_entity_7", "description":"test_entity_7", "createdBy": "admin", "updatedBy": "admin", "attributeDefs":[ { "name":"test_7_1", "isOptional":true, "isUnique":false, "isIndexable":false, "typeName":"string", "valuesMaxCount":1, "cardinality":"SINGLE", "valuesMinCount":0 }, { "name":"test_7_2", "isOptional":true, "isUnique":false, "isIndexable":false, "typeName":"string", "valuesMaxCount":1, "cardinality":"SINGLE", "valuesMinCount":0 } ] } ] } I know the entity is supposed to be in this form, but could someone help me relate this back to my example type please? (I'm not a coder so I have to learn though examples.) POST http://127.0.0.1:21000/api/atlas/v2/entity { "entity" : { "guid" : "...", "status" : "ACTIVE", "createdBy" : "...", "updatedBy" : "...", "createTime" : 12345, "updateTime" : 12345, "version" : 12345, "classifications" : [ { "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } }, { "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } } ], "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } }, "referredEntities" : { "property1" : { "guid" : "...", "status" : "ACTIVE", "createdBy" : "...", "updatedBy" : "...", "createTime" : 12345, "updateTime" : 12345, "version" : 12345, "classifications" : [ { "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } }, { "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } } ], "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } }, "property2" : { "guid" : "...", "status" : "DELETED", "createdBy" : "...", "updatedBy" : "...", "createTime" : 12345, "updateTime" : 12345, "version" : 12345, "classifications" : [ { "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } }, { "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } } ], "typeName" : "...", "attributes" : { "property1" : { }, "property2" : { } } } } } ... View more

Could someone help me understand when I would use the 'options' property? It's part of the AtlasBaseTypeDefs and has a type of 'map of string'. In the below example I can create an entity type, but I don't know how to use the 'options' property and under what circumstances I would want to. { "enumDefs":[], "structDefs":[], "classificationDefs":[], "entityDefs":[ { "superTypes":[ "DataSet" ], "name":"test_entity_3", "description":"test_entity_3", "createdBy": "admin", "updatedBy": "admin", "options" : { "property1" : "when_to_use_1", "property2" : "when_to_use_2" }, "attributeDefs":[ { "name":"test_3", "isOptional":true, "isUnique":false, "isIndexable":false, "typeName":"string", "valuesMaxCount":1, "cardinality":"SINGLE", "valuesMinCount":0 } ] } ] } ... View more

I'm looking at the documentation for HDP 2.6.1 and from what I can see, traits still use the legacy API rather than v2. Am I reading the documentation correctly? In section 'Atlas types' it states that composite metatypes are "Class, Struct, Trait". e.g. POST v2/types/classificationdef POST /v2/types/structdef But v2 REST API doesn't have anything for 'trait' In section 'Cataloging Atlas Metadata: Traits and Business Taxonomy' we have: POST http://<atlas-server-host:port>/api/atlas/types So this is using the legacy API. Is this right? Will this change with future Atlas releases? Also, can someone provide a business use case for using classifications, structs and traits. I'm getting muddled between the circumstances I would use each. Many thanks! ... View more