Created 06-25-2021 04:25 PM
Hello Gurus,
I am having Ranger KMS test connection failed, it is POC test.
CDP 7.1.6 with Isilon OneFS v8.2.2.0, AD kerberos enabled.
Already added following lines in kms-site.xml ( added in Ranger KMS -configuration )
hadoop.kms.proxyuser.rangeradmin.hosts=*
hadoop.kms.proxyuser.rangeradmin.groups=*
hadoop.kms.proxyuser.rangeradmin.users=*
Ranger KMS debug:
2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: ==> RangerPluginClassLoader.deactivate()
2021-06-26 06:51:38,420 DEBUG org.apache.ranger.plugin.classloader.RangerPluginClassLoader: <== RangerPluginClassLoader.deactivate()
2021-06-26 06:51:38,420 ERROR org.apache.hadoop.crypto.key.kms.server.KMS: Exception in getkeyNames.
org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS'
2021-06-26 06:51:38,420 WARN org.apache.hadoop.crypto.key.kms.server.KMS: User ranger (auth:PROXY) via rangeradmin/n02.py.local@PY.LOCAL (auth:KERBEROS) request GET http://n03.py.local:9292/kms/v1/keys/names?doAs=ranger caused exception.
org.apache.hadoop.security.authorize.AuthorizationException: User:ranger not allowed to do 'GET_KEYS'
2021-06-26 06:52:04,559 INFO org.apache.ranger.audit.provider.BaseAuditHandler: Audit Status Log: name=kms.async.summary.multi_dest.batch.solr, interval=01:00.003 minutes, events=1, deferredCount=1, totalEvents=3, totalDeferredCount=3
2021-06-26 06:52:04,560 INFO org.apache.ranger.audit.destination.SolrAuditDestination: Solr zkHosts=null, solrURLs=null, collectionName=ranger_audits
2021-06-26 06:52:04,560 ERROR org.apache.ranger.audit.queue.AuditFileSpool: Error sending logs to consumer. provider=kms.async.summary.multi_dest.batch, consumer=kms.async.summary.multi_dest.batch.solr
2021-06-26 06:52:04,560 INFO org.apache.ranger.audit.queue.AuditFileSpool: Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=kms.async.summary.multi_dest.batch, consumer=kms.async.summary.multi_dest.batch.solr
2021-06-26 06:52:04,691 INFO org.apache.ranger.audit.provider.BaseAuditHandler: Audit Status Log: name=kms.async.summary.multi_dest.batch.hdfs, interval=01:00.012 minutes, events=1, deferredCount=1, totalEvents=3, totalDeferredCount=3
Is there anything mis-configured or need to be checked? Thank you
Best Regards,
Jake Zhang
Created 06-30-2021 01:33 AM
@jakezhang From the screenshot, I can see cm_kms policy is not in sync
Policy needs to be sync after ranger users is added to the policy, then only the Ranger user will be allowed to Get the keys
Created 06-30-2021 07:10 PM
yeah, I was checking the KMS logs, not sure if there is something mis-configured....
2021-07-01 10:03:35,980 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: ==> RangerAdminRESTClient.getServicePoliciesIfUpdated(-1, 1625097035937)
2021-07-01 10:03:35,980 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: Checking Service policy if updated with old api call
2021-07-01 10:03:35,986 DEBUG org.apache.ranger.admin.client.datatype.RESTResponse: fromJson('Unauthenticated access not allowed') failed
org.codehaus.jackson.JsonParseException: Unexpected character ('U' (code 85)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.StringReader@7b831251; line: 1, column: 2]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442)
at org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:1198)
at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:485)
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2770)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2718)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863)
at org.apache.ranger.plugin.util.JsonUtilsV2.jsonToObj(JsonUtilsV2.java:68)
at org.apache.ranger.admin.client.datatype.RESTResponse.fromJson(RESTResponse.java:126)
at org.apache.ranger.admin.client.datatype.RESTResponse.fromClientResponse(RESTResponse.java:100)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:195)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:305)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:244)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:206)
2021-07-01 10:03:35,987 WARN org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies. secureMode=false, user=kms (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=cm_kms
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.admin.client.RangerAdminRESTClient: <== RangerAdminRESTClient.getServicePoliciesIfUpdated(-1, 1625097035937): null
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.plugin.util.PolicyRefresher: PolicyRefresher(serviceName=cm_kms).run(): no update found. lastKnownVersion=-1
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.perf.policyengine.init: [PERF] PolicyRefresher.loadPolicyFromPolicyAdmin(serviceName=cm_kms): 7
2021-07-01 10:03:35,987 DEBUG org.apache.ranger.plugin.util.PolicyRefresher: <== PolicyRefresher(serviceName=cm_kms).loadPolicyfromPolicyAdmin()
Ranger KMS authenticatin type is kerberos, I tired to change it to simple and restarted both ranger and rangerkms, it did not help.
I don't know where the auth simple come from? Thanks.
2021-07-01 10:03:35,987 WARN org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies. secureMode=false, user=kms (auth:SIMPLE), response={"httpStatusCode":400,"statusCode":0}, serviceName=cm_kms
Created 07-06-2021 11:10 AM
I was seeing the same issue, thanks to @jakezhang for posting.
Changing the Ranger KMS: kerberos_princ_name from rangerkms to keyadmin allowed me to get this working. Thanks for the clues in the log file and to @Scharan