question Re: How to create Ranger policy on HDFS path via REST API? in Archives of Support Questions (Read Only)

How to create Ranger policy on HDFS path via REST API?

marcel-jan — Thu, 06 Sep 2018 14:43:15 GMT

I'm trying to roll out new Ranger policies for HDFS with the REST API, but I keep running into errors. We use the Ranger REST API via a Knox gateway, but all that has been proven to work. I was able to use the Ranger REST API to get info about existing policies via the REST API.

I use this command to create a simple policy:

curl -iv -u myaccount -H "Content-Type: application/json" -X POST https://servername:6081/gateway/ui/ranger/service/public/v2/api/policy -d '{ "policyName": "Test: testfile", "resourceName": "/data/test2", "description": "Added automatically via the Ranger REST API", "repositoryName": "ourrepository", "repositoryType": "hdfs", "isEnabled": "true", "isRecursive": "false", "isAuditEnabled": "true", "permMapList": [{ "groupList": ["developers"], "permList": ["Read", "Write", "Execute"] }] }'

And this is the message that I get:

{"statusCode":1,"msgDesc":"(0) Validation failure: error code[3002], reason[Internal error: missing field[service name]], field[service name], subfield[null], type[missing] "}

Any idea what's wrong with my JSON?

Re: How to create Ranger policy on HDFS path via REST API?

JonathanSneep — Thu, 06 Sep 2018 16:03:20 GMT

Hi @Marcel-Jan Krijgsman

Looks like you are missing the "service" value. I get the same [3002] error if I copy-paste your command on my cluster, that error is gone when I add in the "service":"myservice" value (use an existing service, or you'll have to first create a new one).

Re: How to create Ranger policy on HDFS path via REST API?

marcel-jan — Thu, 06 Sep 2018 18:30:43 GMT

I've added the service. Not sure if it just didn't work or that it was followed by a different issue.

 curl -iv -u myaccount -H "Content-Type: application/json"-X POST https://servername:6081/gateway/ui/ranger/service/public/v2/api/policy -d '{ "policyName": "Test: testfile", "service": "OPS_hadoop", "resourceName": "/data/test2", "description": "Added automatically via the Ranger REST API", "repositoryName": "OPS_hadoop", "repositoryType": "hdfs", "isEnabled": "true", "isRecursive": "false", "isAuditEnabled": "true", "permMapList": [{ "groupList": ["developers"], "permList": ["Read", "Write", "Execute"] }] }'

(In my code repositoryName and service are the same BTW).

I got a different error now.

{"statusCode":1,"msgDesc":"(0) Validation failure: error code[3025], reason[Invalid resources specified. hdfs policy must specify values for the following resources: [path] ], field[policy resources], subfield[missing mandatory], type[semantically incorrect] "}

Re: How to create Ranger policy on HDFS path via REST API?

JonathanSneep — Thu, 06 Sep 2018 20:49:11 GMT

Hi @Marcel-Jan Krijgsman
Ok, so it seems we are using the v1 API format while we should be using the v2 format, which is quite different.
I think, the easiest thing to avoid running into the next errors, is to first export the format from an existing policy as an example. Try exporting an existing one first and storing the json;

curl -k -iv -s -u user:pass -H "Content-Type: application/json" -X GET http://myhost:6080/service/public/v2/api/policy/2 > /tmp/exampleformat

With the above, I simply changed the name, path, policyID, uuid value, and posted a new policy.

Then post it back like so;

curl -v -s -ik -u user:pass -H "Content-Type: application/json" -X POST http://myhost:6080/service/public/v2/api/policy -d @exampleformat

This gave me a HTTP/1.1 200 OK & the new policy was visible in the ranger web UI, with the expected config.
Let me know if that's clear. I can attach the exampleformat file I used in the above example if that helps.

Re: How to create Ranger policy on HDFS path via REST API?

marcel-jan — Thu, 06 Sep 2018 21:04:16 GMT

@Jonathan Sneep That worked!

I've edited the json a little. I removed the id and guid from it, but that was no problem. There's some more stuff that probably doesn't have to be in there. I'll try out some more stuff and report back here.

Re: How to create Ranger policy on HDFS path via REST API?

marcel-jan — Thu, 06 Sep 2018 21:19:13 GMT

So you can basically widdle it down to:

{"isEnabled":true,"service":"OPS_hadoop","name":"Test: /data/test2","policyType":0,"description":"Added automatically via the Ranger REST API","isAuditEnabled":true,"resources":{"path":{"values":["/data/test2"],"isExcludes":false,"isRecursive":true}},"policyItems":[{"accesses":[{"type":"read","isAllowed":true},{"type":"write","isAllowed":true},{"type":"execute","isAllowed":true}],"groups":["developers"],"conditions":[],"delegateAdmin":false}],"denyPolicyItems":[],"allowExceptions":[],"denyExceptions":[],"dataMaskPolicyItems":[],"rowFilterPolicyItems":[]}

And this also works.

curl -iv -u 203631 -H "Content-Type: application/json" -X POST https://servername:6801/gateway/ui/ranger/service/public/v2/api/policy -d '{"isEnabled":true,"service":"OPS_hadoop","name":"Test: /data/test2","policyType":0,"description":"Added automatically via the Ranger REST API","isAuditEnabled":true,"resources":{"path":{"values":["/data/test2"],"isExcludes":false,"isRecursive":true}},"policyItems":[{"accesses":[{"type":"read","isAllowed":true},{"type":"write","isAllowed":true},{"type":"execute","isAllowed":true}],"groups":["developers"],"conditions":[],"delegateAdmin":false}],"denyPolicyItems":[],"allowExceptions":[],"denyExceptions":[],"dataMaskPolicyItems":[],"rowFilterPolicyItems":[]}'

Which means I can write a couple of these commands to prepare for a rollout. Cool!

Re: How to create Ranger policy on HDFS path via REST API?

JonathanSneep — Thu, 06 Sep 2018 21:24:15 GMT

Awesome, @Marcel-Jan Krijgsman, glad we got it working 🙂 and thank you for sharing the trimmed result!