CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

[#1]

Hello Gurus,

I am having Ranger KMS test connection failed, it is POC test.

CDP 7.1.6 with Isilon OneFS v8.2.2.0, AD kerberos enabled.

Ranger KMS is up and runningdefault policy login as keyadmintest connection failed

Already added following lines in kms-site.xml ( added in Ranger KMS -configuration )

hadoop.kms.proxyuser.rangeradmin.hosts=* 
hadoop.kms.proxyuser.rangeradmin.groups=* 
hadoop.kms.proxyuser.rangeradmin.users=*

Ranger KMS debug:

2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: ==> RangerPluginClassLoader.deactivate()
2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: <== RangerPluginClassLoader.deactivate()
2021-06-26 06:51:38,420 ERROR org.apache.hadoop.crypto.key.kms.server.KMS: Exception in getkeyNames.
org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS'
2021-06-26 06:51:38,420 WARN org.apache.hadoop.crypto.key.kms.server.KMS: User ranger (auth:PROXY) via rangeradmin/n02.py.local@PY.LOCAL (auth:KERBEROS) request GET http://n03.py.local:9292/kms/v1/keys/names?doAs=ranger caused exception.
org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS'
2021-06-26 06:52:04,559 INFO org.apache.ranger.audit.provider.BaseAuditHandler: Audit Status Log: name=kms.async.summary.multi_dest.batch.solr, interval=01:00.003 minutes, events=1, deferredCount=1, totalEvents=3, totalDeferredCount=3
2021-06-26 06:52:04,560 INFO org.apache.ranger.audit.destination.SolrAuditDestination: Solr zkHosts=null, solrURLs=null, collectionName=ranger_audits
2021-06-26 06:52:04,560 ERROR org.apache.ranger.audit.queue.AuditFileSpool: Error sending logs to consumer. provider=kms.async.summary.multi_dest.batch, consumer=kms.async.summary.multi_dest.batch.solr
2021-06-26 06:52:04,560 INFO org.apache.ranger.audit.queue.AuditFileSpool: Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=kms.async.summary.multi_dest.batch, consumer=kms.async.summary.multi_dest.batch.solr
2021-06-26 06:52:04,691 INFO org.apache.ranger.audit.provider.BaseAuditHandler: Audit Status Log: name=kms.async.summary.multi_dest.batch.hdfs, interval=01:00.012 minutes, events=1, deferredCount=1, totalEvents=3, totalDeferredCount=3

Is there anything mis-configured or need to be checked? Thank you

Best Regards,

Jake Zhang

[#2]

@jakezhang Assign getkeys permission for ranger user in ranger policy

[#3]

Thanks.

However the permissions are already assigned in the default policy: 

  cm_kms

[#4]

@jakezhang Check is  cm_kms policy is in sync 

Also, modify the Config Properties values in cm_kms as shown below 

tag.download.auth.users=kms
policy.download.auth.users=keyadmin,rangerkms

[#5]

Thank you.

You might see they are already added in the previous screenshot.

Ranger user is added as well but it did not work.

tag.download.auth.users=kms,ranger
policy.download.auth.users=keyadmin,rangerkms,ranger

[#6]

@jakezhang  Can you check is cm_kms policy is in sync after adding the ranger users to the policy

Share the screenshot of Ranger Ui => Audit => Plugins

[#7]

Thanks @Scharan 

I don't think it's in sync since the test connection failed.

[#8 · Accepted solution]

@jakezhang From the screenshot, I can see cm_kms policy is not in sync

Policy needs to be sync after ranger users is added to the policy, then only the Ranger user will be allowed to Get the keys

[#9]

Thanks, but how can I get the policy synced?

[#10]

@jakezhang Check ranger KMS logs and see what is the error while refreshing the policy

[#11]

yeah, I was checking the KMS logs, not sure if there is something mis-configured....

2021-07-01 10:03:35,980 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: ==> RangerAdminRESTClient.getServicePoliciesIfUpdated(-1, 1625097035937)
2021-07-01 10:03:35,980 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: Checking Service policy if updated with old api call
2021-07-01 10:03:35,986 DEBUG org.apache.ranger.admin.client.datatype.RESTResponse: fromJson('Unauthenticated access not allowed') failed
org.codehaus.jackson.JsonParseException: Unexpected character ('U' (code 85)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@7b831251; line: 1, column: 2]
        at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
        at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
        at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442)
        at org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:1198)
        at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:485)
        at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2770)
        at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2718)
        at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863)
        at org.apache.ranger.plugin.util.JsonUtilsV2.jsonToObj(JsonUtilsV2.java:68)
        at org.apache.ranger.admin.client.datatype.RESTResponse.fromJson(RESTResponse.java:126)
        at org.apache.ranger.admin.client.datatype.RESTResponse.fromClientResponse(RESTResponse.java:100)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:195)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:305)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:244)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:206)
2021-07-01 10:03:35,987 WARN org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies. secureMode=false, user=kms (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=cm_kms
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: <== RangerAdminRESTClient.getServicePoliciesIfUpdated(-1, 1625097035937): null
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.plugin.util.PolicyRefresher: PolicyRefresher(serviceName=cm_kms).run(): no update found. lastKnownVersion=-1
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.perf.policyengine.init: [PERF] PolicyRefresher.loadPolicyFromPolicyAdmin(serviceName=cm_kms): 7
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.plugin.util.PolicyRefresher: <== PolicyRefresher(serviceName=cm_kms).loadPolicyfromPolicyAdmin()

Ranger KMS authenticatin type is kerberos, I tired to change it to simple and restarted both ranger and rangerkms, it did not help.

I don't know where the auth simple come from? Thanks.

2021-07-01 10:03:35,987 WARN org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies. secureMode=false, user=kms (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=cm_kms

[#12]

I was seeing the same issue, thanks to @jakezhang  for posting.

Changing the Ranger KMS: kerberos_princ_name from rangerkms to keyadmin allowed me to get this working. Thanks for the clues in the log file and to @Scharan