Nifi / Ranger / Audit to Solr / unable to find valid certification path to requested target

[#1]

Hello,

I have a HDF cluster (kerberos/ ranger) on which the client nodes all reports the following

2017-03-09 14:54:00,662 WARN [org.apache.ranger.audit.queue.AuditBatchQueue0] o.a.r.audit.provider.BaseAuditHandler failed to log audit event: {"repoType":10,"repo":"<MASKED>","reqUser":"<MASKED>","evtTime":"2017-03-09 14:54:00.275","access":"WRITE","resource":"/proxy","resType":"nifi-resource","action":"WRITE","result":1,"policy":2,"enforcer":"ranger-acl","cliIP":"<MASKED>","agentHost":"<MASKED>","logType":"RangerAudit","id":"56f7f5c4-a834-4405-9bae-18b19453129d-140","seq_num":276,"event_count":1,"event_dur_ms":0,"tags":[]}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException occured when talking to server at: https://<Ambari_solr_FQDN>:8886/solr/ranger_audits_shard1_replica1
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:634) ~[solr-solrj-5.5.1.jar:5.5.1
				<SNIP>
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://<Ambari_solr_FQDN>:8886/solr/ranger_audits_shard1_replica1
        <SNIP>
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        <SNIP>
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        <SNIP>
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        <SNIP>

I have checked the keystore & truststore storead at the location refered in Ambari: the chain looks correct to me.

curl https://<Ambari_solr_FQDN>:8886/solr/ranger_audits_shard1_replica1 connects without errors

openssl indicates a valid chain.

I can't find which store / chain is used by Nifi and how to fix this.

Any help will be welcome.

Thanks

Christophe

[#2]

@Christophe Vico

This appears to be an issue between Ranger and Solr and have nothing to do with NiFi at all. I suggest updating your tags on this post to include Solr.

[#3]

@Matt Clarke thanks. I update the tags, I was not too sure actually where to submit this.

[#4 · Accepted solution]

I believe this is an issue between the truststore used by the plugin and certificates used by Solr... basically the SSL handshake believes that the certificate Solr is using is not trusted by whatever is in the truststore that the plugin is using.

How did you enable SSL on Solr? Did you generate your own certificate and do this manually?

The Ranger plugin that runs inside the NiFi JVM process (which is what sends the audits to Solr) will use the values of xasecure.policymgr.clientssl.truststore, xasecure.policymgr.clientssl.truststore.password, and xasecure.policymgr.clientssl.truststore.credential.file which come from ranger-nifi-policymgr-ssl.xml

So the truststore specified there needs to trust the certificate authority that created the cert that Solr is using.

Also, this issue could be related, but not sure:

https://issues.apache.org/jira/browse/RANGER-1216

Looks like it was fixed for Ranger 0.7, but I believe HDF is using 0.6.x.

[#5]

@bryan bende

Thanks for answers.

The truststore & keystore listed in the Nifi configuration (xasecure.policymgr.clientssl.*) are the one I checked, containing the right certificates as far a I can tell. The trustore.jks does contain the root CA used to issue the certificates

I have again rechecked, and made sure that nifi:hadoop was onwer of the stores, but to no luck.

I don't think the JIRa is linked, as in my case, I don;t establish the SSL connection, so I can't possibly yet be impacted by Kerberos

Thanks!

[#6]

Hello,

I found the cause of this one : the keystore was specified as truststore for Ranger plugin. I missed it while reviewing the configs.

Thanks @Bryan Bende!

[#7]

Glad you got it working!