I'm trying to roll out new Ranger policies for HDFS with the REST API, but I keep running into errors. We use the Ranger REST API via a Knox gateway, but all that has been proven to work. I was able to use the Ranger REST API to get info about existing policies via the REST API. I use this command to create a simple policy: curl -iv -u myaccount -H "Content-Type: application/json" -X POST https://servername:6081/gateway/ui/ranger/service/public/v2/api/policy -d '{ "policyName": "Test: testfile", "resourceName": "/data/test2", "description": "Added automatically via the Ranger REST API", "repositoryName": "ourrepository", "repositoryType": "hdfs", "isEnabled": "true", "isRecursive": "false", "isAuditEnabled": "true", "permMapList": [{ "groupList": ["developers"], "permList": ["Read", "Write", "Execute"] }] }' And this is the message that I get: {"statusCode":1,"msgDesc":"(0) Validation failure: error code[3002], reason[Internal error: missing field[service name]], field[service name], subfield[null], type[missing] "} Any idea what's wrong with my JSON? ... View more

I have a couple of external Hive tables of which I need to gain a group of users access to only the non sensitive columns with Ranger (HDP 2.6.3). But during some tests with a testuser I found that he can only access these non sensitive columns if he has access to the path on HDFS. The HDFS path is secured by Ranger as well. I've set the HDFS permissions to no access at all. Needing HDFS access for those users would defeat te purpose of granting access to a selection of columns. Because the HDFS file would contain all the sensitive data as well. Is there a way to protect both the HDFS files and grant access to a selection of columns? Example create table statement: CREATE EXTERNAL TABLE berth_data (`MUTATION_TYPE` string, `D_MUTATION` string, `T_MUTATION` string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION '/data/production/sensitive/berth_data'; ... View more

On a HDP 2.6.3 sandbox I'm creating a couple of HDFS path entities in bulk. This is the JSON file I've used: { "entities": [ { "typeName": "hdfs_path", "attributes": { "path": "/user/dmaster/electionresults", "qualifiedName": "hdfs://sandbox.hortonworks.com:8020/user/dmaster/electionresults", "name": "/user/dmaster/electionresults" }, "classification": [], "status": "ACTIVE" }, { "typeName": "hdfs_path", "attributes": { "path": "/user/dmaster/nyse", "qualifiedName": "hdfs://sandbox.hortonworks.com:8020/user/dmaster/nyse", "name": "/user/dmaster/nyse" }, "classification": [], "status": "ACTIVE" }, { "typeName": "hdfs_path", "attributes": { "path": "/user/dmaster/lca", "qualifiedName": "hdfs://sandbox.hortonworks.com:8020/user/dmaster/lca", "name": "/user/dmaster/lca" }, "classification": [], "status": "ACTIVE" } ], "referredEntities": {} } And this is the Atlas REST API call I've ran: curl -X POST -d @atlas_create_entities_bulk.json -u holger_gov:holger_gov -H 'Content-Type: application/json; charset=UTF-8' http://sandbox.hortonworks.com:21000/api/atlas/v2/entity/bulk This resulted after a while in this error: {"errorCode":"ATLAS-500-00-007","errorMessage":"Failed to notify for change CREATE"} The HDFS path entities have been created (quite quickly I saw) and they seem okay. Should I be worried about this error message? Here are the details on the created entities, retrieved with a /v2/search/dsl API call: { "typeName": "hdfs_path", "attributes": { "owner": null, "qualifiedName": "hdfs://sandbox.hortonworks.com:8020/user/dmaster/electionresults", "name": "/user/dmaster/electionresults", "description": null }, "guid": "b81c8a22-0346-4261-8cb7-1b0be91ee134", "status": "ACTIVE", "displayText": "/user/dmaster/electionresults", "classificationNames": [] }, { "typeName": "hdfs_path", "attributes": { "owner": null, "qualifiedName": "hdfs://sandbox.hortonworks.com:8020/user/dmaster/nyse", "name": "/user/dmaster/nyse", "description": null }, "guid": "26b81bb8-4dcb-48d0-b0d4-37df38fff1b2", "status": "ACTIVE", "displayText": "/user/dmaster/nyse", "classificationNames": [] }, { "typeName": "hdfs_path", "attributes": { "owner": null, "qualifiedName": "hdfs://sandbox.hortonworks.com:8020/user/dmaster/lca", "name": "/user/dmaster/lca", "description": null }, "guid": "e4789fba-67ef-4195-bc3f-afb1623985f8", "status": "ACTIVE", "displayText": "/user/dmaster/lca", "classificationNames": [] } ... View more