Created 03-09-2017 02:39 PM
Hello,
I have a HDF cluster (kerberos/ ranger) on which the client nodes all reports the following
2017-03-09 14:54:00,662 WARN [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.audit.provider.BaseAuditHandler failed to log audit event: {"repoType":10,"repo":"<MASKED>","reqUser":"<MASKED>","evtTime":"2017-03-09 14:54:00.275","access":"WRITE","resource":"/proxy","resType":"nifi-resource","action":"WRITE","result":1,"policy":2,"enforcer":"ranger-acl","cliIP":"<MASKED>","agentHost":"<MASKED>","logType":"RangerAudit","id":"56f7f5c4-a834-4405-9bae-18b19453129d-140","seq_num":276,"event_count":1,"event_dur_ms":0,"tags":[]} org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException occured when talking to server at: https://<Ambari_solr_FQDN>:8886/solr/ranger_audits_shard1_replica1 at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:634) ~[solr-solrj-5.5.1.jar:5.5.1 <SNIP> Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://<Ambari_solr_FQDN>:8886/solr/ranger_audits_shard1_replica1 <SNIP> Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target <SNIP> Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target <SNIP> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target <SNIP>
I have checked the keystore & truststore storead at the location refered in Ambari: the chain looks correct to me.
curl https://<Ambari_solr_FQDN>:8886/solr/ranger_audits_shard1_replica1 connects without errors
openssl indicates a valid chain.
I can't find which store / chain is used by Nifi and how to fix this.
Any help will be welcome.
Thanks
Christophe
Created 03-10-2017 05:37 PM
I believe this is an issue between the truststore used by the plugin and certificates used by Solr... basically the SSL handshake believes that the certificate Solr is using is not trusted by whatever is in the truststore that the plugin is using.
How did you enable SSL on Solr? Did you generate your own certificate and do this manually?
The Ranger plugin that runs inside the NiFi JVM process (which is what sends the audits to Solr) will use the values of xasecure.policymgr.clientssl.truststore, xasecure.policymgr.clientssl.truststore.password, and xasecure.policymgr.clientssl.truststore.credential.file which come from ranger-nifi-policymgr-ssl.xml
So the truststore specified there needs to trust the certificate authority that created the cert that Solr is using.
Also, this issue could be related, but not sure:
https://issues.apache.org/jira/browse/RANGER-1216
Looks like it was fixed for Ranger 0.7, but I believe HDF is using 0.6.x.
Created 03-09-2017 03:11 PM
This appears to be an issue between Ranger and Solr and have nothing to do with NiFi at all. I suggest updating your tags on this post to include Solr.
Created 03-09-2017 03:34 PM
@Matt Clarke thanks. I update the tags, I was not too sure actually where to submit this.
Created 03-10-2017 05:37 PM
I believe this is an issue between the truststore used by the plugin and certificates used by Solr... basically the SSL handshake believes that the certificate Solr is using is not trusted by whatever is in the truststore that the plugin is using.
How did you enable SSL on Solr? Did you generate your own certificate and do this manually?
The Ranger plugin that runs inside the NiFi JVM process (which is what sends the audits to Solr) will use the values of xasecure.policymgr.clientssl.truststore, xasecure.policymgr.clientssl.truststore.password, and xasecure.policymgr.clientssl.truststore.credential.file which come from ranger-nifi-policymgr-ssl.xml
So the truststore specified there needs to trust the certificate authority that created the cert that Solr is using.
Also, this issue could be related, but not sure:
https://issues.apache.org/jira/browse/RANGER-1216
Looks like it was fixed for Ranger 0.7, but I believe HDF is using 0.6.x.
Created 03-10-2017 07:48 PM
@bryan bende
Thanks for answers.
The truststore & keystore listed in the Nifi configuration (xasecure.policymgr.clientssl.*) are the one I checked, containing the right certificates as far a I can tell. The trustore.jks does contain the root CA used to issue the certificates
I have again rechecked, and made sure that nifi:hadoop was onwer of the stores, but to no luck.
I don't think the JIRa is linked, as in my case, I don;t establish the SSL connection, so I can't possibly yet be impacted by Kerberos
Thanks!
Created 03-13-2017 04:18 PM
Hello,
I found the cause of this one : the keystore was specified as truststore for Ranger plugin. I missed it while reviewing the configs.
Thanks @Bryan Bende!
Created 03-13-2017 04:30 PM
Glad you got it working!