How to create process-groups and apply custom policies to the processor via Nifi API

[#1]

Hi All

I have a list of users that i have in an array.

I am trying to create a job in shell script that creates a process-group for each user and apply policy to that particular process group so that only that user and nifi's super user can access or operate in it 

here is how the polices config json looks like 

'[{"globalAccessPolicies":[{"resource":"flow","action":"read"},{"resource":"provenance","action":"read"}],"componentLevelAccessPolicies":[{"resource":"process-groups","action":"read"},{"resource":"process-groups","action":"write"},{"resource":"operation/process-groups","action":"write"},{"resource":"provenance-data/process-groups","action":"write"},{"resource":"provenance-data/process-groups","action":"read"},{"resource":"data/process-groups","action":"read"},{"resource":"data/process-groups","action":"write"},{"resource":"policies/process-groups","action":"read"},{"resource":"policies/process-groups","action":"write"},{"resource":"data-transfer/input-ports","action":"write"},{"resource":"data-transfer/output-ports","action":"write"}]}]'     

For each user in the array, the job iterates through globalAccessPolicies and componentLevelAccessPolicies and assigns permission to the user.

before assigning the componentLevelAccessPolicies the job creates an empty process-group to in the nifi canvas 

using the api 

nifi_api_request "process-groups/root/process-groups" "POST" "Content-Type: application/json" "{\"revision\":{\"version\":0},\"component\":{\"name\":\"${tenant}\",\"position\":{\"x\":${x},\"y\":${y}},\"comments\":\"Processor group for ${tenant}\"}}"

this returns a json from which the id of the process group is fetched 

then the Job uses the fetched ID, uses the componentLevelAccessPolicies array and tries to create a policy configuration using 

for policy in $(echo "${componentLevelAccessPolicies}" | jq -c '.[]'); do
                      resource=$(echo "${policy}" | jq -r '.resource')
                      action=$(echo "${policy}" | jq -r '.action')
                      policyConfig=$(echo '{
                                          "revision": {
                                            "version": 0
                                          },
                                          "component": {
                                            "resource": "'${resource}'",
                                            "action": "'${action}'",
                                            "configurable": true,
                                            "users": [],
                                            "userGroups": []
                                          }
                                        }' | jq .)

                      policyConfig=$(echo "${policyConfig}" | jq ".component.userGroups += [{\"revision\":{\"version\":0},\"id\":\"${tenant_superadmin_id}\",\"permissions\":{\"canRead\":true,\"canWrite\":true},\"component\":{\"id\":\"${processor_group_id}\",\"identity\":\"${tenant_superadmin}\",\"configurable\":true}}]")
                      policyConfig=$(echo "${policyConfig}" | jq ".component.userGroups += [{\"revision\":{\"version\":0},\"id\":\"${tenant_readonly_id}\",\"permissions\":{\"canRead\":true,\"canWrite\":false},\"component\":{\"id\":\"${processor_group_id}\",\"identity\":\"${tenant_readonly}\",\"configurable\":true}}]")
                      policyConfig=$(echo "${policyConfig}" | jq ".component.userGroups += [{\"revision\":{\"version\":0},\"id\":\"${nifi_superadmin_id}\",\"permissions\":{\"canRead\":true,\"canWrite\":true},\"component\":{\"id\":\"${processor_group_id}\",\"identity\":\"${nifi_superadmin}\",\"configurable\":true}}]")
                      policyConfig=$(echo "${policyConfig}" | jq ".component.userGroups += [{\"revision\":{\"version\":0},\"id\":\"${nifi_readonly_id}\",\"permissions\":{\"canRead\":true,\"canWrite\":false},\"component\":{\"id\":\"${processor_group_id}\",\"identity\":\"${nifi_readonly}\",\"configurable\":true}}]")

                      echo "policy Group Config: ${policyConfig}"
                      nifi_api_request "policies/" "POST" "Content-Type: application/json" "${policyConfig}"
                      echo -e "status ${status}"
                      echo -e "body ${body}"
                      [[ ${status} -eq 201 ]] || exit 1

                    done

The required Ids are fetched beforehand from the user groups 

 Since job is trying to create policy to each processor group, i have set  the http method as POST

      nifi_api_request "policies/" "POST" "Content-Type: application/json" "${policyConfig}"
 

in the Loop it creates a json paylod like this 

{
"revision": {
"version": 0
},
"component": {
"resource": "data-transfer/output-ports",
"action": "write",
"configurable": true,
"users": [],
"userGroups": [
{
"revision": {
"version": 0
},
"id": "9fd4eabd-5b6f-4a1d-8c5f-ca6049986d96",
"permissions": {
"canRead": true,
"canWrite": true
},
"component": {
"id": "9ef8a5e3-0195-1000-ffff-ffffb4a7b545",
"identity": "TenantID.john.nifi_superadmin",
"configurable": true
}
},
{
"revision": {
"version": 0
},
"id": "aba0f614-d09a-42ee-9081-3328c86fcd6e",
"permissions": {
"canRead": true,
"canWrite": false
},
"component": {
"id": "9ef8a5e3-0195-1000-ffff-ffffb4a7b545",
"identity": "TenantID.john.nifi_readonly",
"configurable": true
}
},
{
"revision": {
"version": 0
},
"id": "ebe9c88a-77d7-4070-bead-e24329b2e9c1",
"permissions": {
"canRead": true,
"canWrite": true
},
"component": {
"id": "9ef8a5e3-0195-1000-ffff-ffffb4a7b545",
"identity": "nifi_superadmin",
"configurable": true
}
},
{
"revision": {
"version": 0
},
"id": "95aedc49-20f6-4dad-95f5-c7311a66c353",
"permissions": {
"canRead": true,
"canWrite": false
},
"component": {
"id": "9ef8a5e3-0195-1000-ffff-ffffb4a7b545",
"identity": "nifi_readonly",
"configurable": true
}
}
]
}
}
 ​

the job did not throw any error for the first user in the loop ( though the process-group did not have the users assigned in the canvas PFA).

but when it run for the second user it returns 400 bad request 

status 409
body Found multiple policies for 'process-groups' with 'read'.
​

Note:

I have tried with configurations by setting the user-group id in the policy config json 

policyConfig=$(echo "${policyConfig}" | jq ".component.userGroups += [{\"revision\":{\"version\":0},\"id\":\"${tenant_superadmin_id}\",\"permissions\":{\"canRead\":true,\"canWrite\":true},\"component\":{\"id\":\"${tenant_superadmin_id}\",\"identity\":\"${tenant_superadmin}\",\"configurable\":true}}]")

This did not work, because processor group id is not linked

Can anyone please advise to how to set the policies uniquely for each process group so that only the user and super admins can access it?

From the canvas it is achievable, when i select override as empty, But i need this to be done via API call  

Thanks much for your time.

[#2]

Note 2:

I have also tried to fetch the policy id from the processor and appending it a policy configuration 

using 

nifi_api_request "policies/${action}${resource}" "GET"

example

policies/read/data/process-groups/9486a139-0195-1000-0000-00000ddc9b4f

gives a json 

job will fetch the ID of the policy 

the call the policy 

nifi_api_request "policies/${policyId}" "GET"

use policyConfig to append info where entity_id is the id of the user and entity_name is the name

policyConfig=$(echo ${body} | jq ".component.userGroups[.component.userGroups | length] |= {\"revision\":{\"version\":0},\"id\":\"${entity_id}\",\"permissions\":{\"canRead\":${canRead},\"canWrite\":${canWrite}},\"component\":{\"id\":\"${entity_id}\",\"identity\":\"${entity_name}\",\"configurable\":true}}")

This unfortunately started created large json and ended up assigning all the users access to all the process-groups 

[#3]

Update :

i realized that I wasn't adding process-group id to the resources, so i added the process-group to the policy config json 

policy Group Config: {
  "revision": {
    "version": 0
  },
  "permissions": {
    "canRead": true,
    "canWrite": true
  },
  "component": {
    "resource": "data-transfer/output-ports/a2a202da-0195-1000-0000-000045d2086d",
    "action": "write",
    "configurable": true,
    "users": [],
    "userGroups": [
      {
        "revision": {
          "version": 0
        },
        "id": "3d872ce1-e08a-4794-85e5-66e5a1f2f4ac",
        "permissions": {
          "canRead": true,
          "canWrite": true
        },
        "component": {
          "id": "3d872ce1-e08a-4794-85e5-66e5a1f2f4ac",
          "identity": "TenantID.john.nifi_superadmin",
          "configurable": true
        }
      },
      {
        "revision": {
          "version": 0
        },
        "id": "ebe9c88a-77d7-4070-bead-e24329b2e9c1",
        "permissions": {
          "canRead": true,
          "canWrite": true
        },
        "component": {
          "id": "ebe9c88a-77d7-4070-bead-e24329b2e9c1",
          "identity": "nifi_superadmin",
          "configurable": true
        }
      }
    ]
  }
}

"data-transfer/output-ports/a2a202da-0195-1000-0000-000045d2086d",

where a2a202da-0195-1000-0000-000045d2086d is the id of the process group

all the policies were created with 201

when i try to access the policies of the process group , i don't see the respective user-group having privilege to it, it just has super admins in the list

[#4 · Accepted solution]

Update :

Solved it 

i was missing a '/' infront of the resources that i was providing 

it should have been 

"resource": "/data-transfer/output-ports/a2a202da-0195-1000-0000-000045d2086d",

instead of 

"resource": "data-transfer/output-ports/a2a202da-0195-1000-0000-000045d2086d",

comparing the policies created with UI and with API made me realize it