question Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' " in Support Questions

CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

jakezhang — Fri, 25 Jun 2021 23:25:46 GMT

Hello Gurus,

I am having Ranger KMS test connection failed, it is POC test.

CDP 7.1.6 with Isilon OneFS v8.2.2.0, AD kerberos enabled.

Ranger KMS is up and runningdefault policy login as keyadmintest connection failed

Already added following lines in kms-site.xml ( added in Ranger KMS -configuration )

hadoop.kms.proxyuser.rangeradmin.hosts=* hadoop.kms.proxyuser.rangeradmin.groups=* hadoop.kms.proxyuser.rangeradmin.users=*

Ranger KMS debug:

2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: ==> RangerPluginClassLoader.deactivate() 2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: <== RangerPluginClassLoader.deactivate() 2021-06-26 06:51:38,420 ERROR org.apache.hadoop.crypto.key.kms.server.KMS: Exception in getkeyNames. org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS' 2021-06-26 06:51:38,420 WARN org.apache.hadoop.crypto.key.kms.server.KMS: User ranger (auth:PROXY) via rangeradmin/n02.py.local@PY.LOCAL (auth:KERBEROS) request GET http://n03.py.local:9292/kms/v1/keys/names?doAs=ranger caused exception. org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS' 2021-06-26 06:52:04,559 INFO org.apache.ranger.audit.provider.BaseAuditHandler: Audit Status Log: name=kms.async.summary.multi_dest.batch.solr, interval=01:00.003 minutes, events=1, deferredCount=1, totalEvents=3, totalDeferredCount=3 2021-06-26 06:52:04,560 INFO org.apache.ranger.audit.destination.SolrAuditDestination: Solr zkHosts=null, solrURLs=null, collectionName=ranger_audits 2021-06-26 06:52:04,560 ERROR org.apache.ranger.audit.queue.AuditFileSpool: Error sending logs to consumer. provider=kms.async.summary.multi_dest.batch, consumer=kms.async.summary.multi_dest.batch.solr 2021-06-26 06:52:04,560 INFO org.apache.ranger.audit.queue.AuditFileSpool: Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=kms.async.summary.multi_dest.batch, consumer=kms.async.summary.multi_dest.batch.solr 2021-06-26 06:52:04,691 INFO org.apache.ranger.audit.provider.BaseAuditHandler: Audit Status Log: name=kms.async.summary.multi_dest.batch.hdfs, interval=01:00.012 minutes, events=1, deferredCount=1, totalEvents=3, totalDeferredCount=3

Is there anything mis-configured or need to be checked? Thank you

Best Regards,

Jake Zhang

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

Scharan — Sat, 26 Jun 2021 15:54:34 GMT

@jakezhang Assign getkeys permission for ranger user in ranger policy

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

jakezhang — Sun, 27 Jun 2021 00:59:21 GMT

Thanks.

However the permissions are already assigned in the default policy:

cm_kms

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

Scharan — Sun, 27 Jun 2021 16:42:31 GMT

@jakezhang Check is cm_kms policy is in sync

Also, modify the Config Properties values in cm_kms as shown below

tag.download.auth.users=kms policy.download.auth.users=keyadmin,rangerkms

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

jakezhang — Tue, 29 Jun 2021 22:26:01 GMT

Thank you.

You might see they are already added in the previous screenshot.

Ranger user is added as well but it did not work.

tag.download.auth.users=kms,ranger policy.download.auth.users=keyadmin,rangerkms,ranger

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

Scharan — Wed, 30 Jun 2021 05:02:47 GMT

@jakezhang Can you check is cm_kms policy is in sync after adding the ranger users to the policy

Share the screenshot of Ranger Ui => Audit => Plugins

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

jakezhang — Wed, 30 Jun 2021 08:19:39 GMT

Thanks @Scharan

I don't think it's in sync since the test connection failed.

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

Scharan — Wed, 30 Jun 2021 08:33:23 GMT

@jakezhang From the screenshot, I can see cm_kms policy is not in sync

Policy needs to be sync after ranger users is added to the policy, then only the Ranger user will be allowed to Get the keys

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

jakezhang — Wed, 30 Jun 2021 22:05:35 GMT

Thanks, but how can I get the policy synced?

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

Scharan — Thu, 01 Jul 2021 00:51:53 GMT

@jakezhang Check ranger KMS logs and see what is the error while refreshing the policy

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

jakezhang — Thu, 01 Jul 2021 02:10:30 GMT

yeah, I was checking the KMS logs, not sure if there is something mis-configured....

2021-07-01 10:03:35,980 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: ==> RangerAdminRESTClient.getServicePoliciesIfUpdated(-1, 1625097035937) 2021-07-01 10:03:35,980 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: Checking Service policy if updated with old api call 2021-07-01 10:03:35,986 DEBUG org.apache.ranger.admin.client.datatype.RESTResponse: fromJson('Unauthenticated access not allowed') failed org.codehaus.jackson.JsonParseException: Unexpected character ('U' (code 85)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: java.io.StringReader@7b831251; line: 1, column: 2] at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433) at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521) at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442) at org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:1198) at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:485) at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2770) at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2718) at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863) at org.apache.ranger.plugin.util.JsonUtilsV2.jsonToObj(JsonUtilsV2.java:68) at org.apache.ranger.admin.client.datatype.RESTResponse.fromJson(RESTResponse.java:126) at org.apache.ranger.admin.client.datatype.RESTResponse.fromClientResponse(RESTResponse.java:100) at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:195) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:305) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:244) at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:206) 2021-07-01 10:03:35,987 WARN org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies. secureMode=false, user=kms (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=cm_kms 2021-07-01 10:03:35,987 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: <== RangerAdminRESTClient.getServicePoliciesIfUpdated(-1, 1625097035937): null 2021-07-01 10:03:35,987 DEBUG org.apache.ranger.plugin.util.PolicyRefresher: PolicyRefresher(serviceName=cm_kms).run(): no update found. lastKnownVersion=-1 2021-07-01 10:03:35,987 DEBUG org.apache.ranger.perf.policyengine.init: [PERF] PolicyRefresher.loadPolicyFromPolicyAdmin(serviceName=cm_kms): 7 2021-07-01 10:03:35,987 DEBUG org.apache.ranger.plugin.util.PolicyRefresher: <== PolicyRefresher(serviceName=cm_kms).loadPolicyfromPolicyAdmin()

Ranger KMS authenticatin type is kerberos, I tired to change it to simple and restarted both ranger and rangerkms, it did not help.

I don't know where the auth simple come from? Thanks.

2021-07-01 10:03:35,987 WARN org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies. secureMode=false, user=kms (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=cm_kms

Re: CDP 7.1.6 Ranger KMS test conection failed " User:ranger not allowed to do 'GET_KEYS' "

russ_stevenson — Tue, 06 Jul 2021 18:10:47 GMT

I was seeing the same issue, thanks to @jakezhang for posting.

Changing the Ranger KMS: kerberos_princ_name from rangerkms to keyadmin allowed me to get this working. Thanks for the clues in the log file and to @Scharan