Support Questions

ollie1 · ‎01-18-2017

I'm having a torrid time trying to configure ranger with NiFi, with both services setup with SSL already. I've been following this guide: https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range....

It was previously working without SSL, so something is wrong with my keystore's and truststores. I'll describe my setup in as much detail as possible, and I'm hoping, between my config, and logs, we can make some progress debugging the issue.

Overview:

Ambari (HDF-2.1.1.0) is managing a ranger and NIFI install, all on separate instances (ambari-1, nifi-1 and ranger-1). I have not configured a NIFI Certificate Authority.

NiFi instance:

Truststores:

/etc/security/nifi-certs/keystore.jks /etc/security/nifi-certs/truststore.jks

/etc/security/ranger-certs/keystore.jks /etc/security/ranger-certs/truststore.jks

nifi.properties

nifi.security.identity.mapping.pattern.dn=
nifi.security.identity.mapping.pattern.kerb=
nifi.security.identity.mapping.value.dn=
nifi.security.identity.mapping.value.kerb=
nifi.security.keyPasswd=easypass
nifi.security.keyPasswd.protected=aes/gcm/256
nifi.security.keystore=/etc/security/nifi-certs/keystore.jks
nifi.security.keystorePasswd=easypass
nifi.security.keystorePasswd.protected=aes/gcm/256
nifi.security.keystoreType=JKS
nifi.security.needClientAuth=False
nifi.security.ocsp.responder.certificate=
nifi.security.ocsp.responder.url=
nifi.security.truststore=/etc/security/nifi-certs/truststore.jks
nifi.security.truststorePasswd=easypass
nifi.security.truststorePasswd.protected=aes/gcm/256
nifi.security.truststoreType=JKS
nifi.security.user.authorizer=ranger-provider
nifi.security.user.login.identity.provider=kerberos-provider

ranger-policymgr-ssl.xml

 <configuration>
    <property>
      <name>owner.for.certificate</name>
      <value></value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.keystore</name>
      <value>/etc/security/nifi-certs/keystore.jks</value>
    </property>
   <property>
      <name>xasecure.policymgr.clientssl.keystore.credential.file</name>
      <value>jceks://file/etc/ranger/NiFi_nifi/cred.jceks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.keystore.password</name>
      <value>easypass</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore</name>
      <value>/etc/security/nifi-certs/truststore.jks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore.credential.file</name>
      <value>jceks://file/etc/ranger/NiFi_nifi/cred.jceks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore.password</name>
      <value>easypass</value>
    </property>
  </configuration>

No notable ERROR messages appearing in nifi-app.log

Ranger instance:

Truststores:

/etc/security/ranger-certs/keystore.jks /etc/security/ranger-certs/truststore.jks

ranger-admin-site.xml

   <property>
      <name>ranger.truststore.file</name>
      <value>/etc/security/ranger-certs/truststore.jks</value>
    </property>
    <property>
      <name>ranger.truststore.password</name>
      <value>easypass</value>
    </property>
    <property>
      <name>ranger.https.attrib.keystore.file</name>
      <value>/etc/security/ranger-certs/keystore.jks</value>
    </property>
  <property>
      <name>ranger.service.https.attrib.keystore.keyalias</name>
      <value>ranger-1</value>
    </property>
    <property>
      <name>ranger.service.https.attrib.keystore.pass</name>
      <value>easypass</value>
    </property>

Error logs (xa_portal.log) are showing that one of my keystore's password's is incorrect:

2017-01-18 19:40:54,646 [timed-executor-pool-0] ERROR org.apache.ranger.services.nifi.RangerServiceNiFi (RangerServiceNiFi.java:51) - <== RangerServiceNiFi.validateConfig Error:
java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.createSslContext(NiFiConnectionMgr.java:138)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.getNiFiClient(NiFiConnectionMgr.java:92)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.connectionTest(NiFiConnectionMgr.java:106)
        at org.apache.ranger.services.nifi.RangerServiceNiFi.validateConfig(RangerServiceNiFi.java:49)
        at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
        at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
        at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)

Though I've not been able to deduce which keystore this is complaining about!

and another REST ERROR

2017-01-18 20:03:45,901 [ranger-1.nifi.local-startStop-1] ERROR org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:138) - EmbeddedServiceDefsUtil.init(): failed
javax.ws.rs.WebApplicationException
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:311)
        at org.apache.ranger.service.RangerBaseModelService.read(RangerBaseModelService.java:234)
        at org.apache.ranger.biz.ServiceDBStore.getServiceDef(ServiceDBStore.java:1264)
        at org.apache.ranger.plugin.store.AbstractServiceStore.updateTagServiceDefForUpdatingAccessTypes(AbstractServiceStore.java:297)
        at org.apache.ranger.plugin.store.AbstractServiceStore.updateTagServiceDefForAccessTypes(AbstractServiceStore.java:55)
        at org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil.init(EmbeddedServiceDefsUtil.java:136)
        at org.apache.ranger.biz.ServiceDBStore$1.doInTransaction(ServiceDBStore.java:287)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at org.apache.ranger.biz.ServiceDBStore.initStore(ServiceDBStore.java:284)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Me

Thank's in advance for any help.

EDIT 1:

EDIT 2:

Step 1 & 2 in the guide:

[root@nifi-1 nifi-certs]# keytool -list -keystore truststore.jks
Enter keystore password:


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 2 entries


rootca, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 80:60:76:CF:8B:ED:37:79:73:3A:03:28:B3:9E:A9:AE:E9:03:EF:CD
mykey, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 9E:39:B3:8E:B3:37:76:2F:E5:99:CC:D1:13:E6:71:FC:1A:F1:C9:C8
[root@nifi-1 nifi-certs]#

Step 3 & 4:

[root@ranger-1 security]# cd /etc/security/ranger-certs/
[root@ranger-1 ranger-certs]# keytool -list -keystore truststore.jks
Enter keystore password:


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 1 entry


nifi-1, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 9C:52:46:2D:90:3E:B7:24:D3:3F:0E:E4:21:DD:D6:0B:28:74:70:E4
[root@ranger-1 ranger-certs]#

EDIT 3:

Revised key and trust stores as @Yolanda M. Davis advised.

Errors above have stopped on the ranger node, and started on the NiFi.

2017-01-18 22:09:59,406 WARN [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-1.nifi.local due to javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_77]
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) ~[na:1.8.0_77]
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_77]
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:71) ~[na:1.8.0_77]
 at org.apache.nifi.cluster.protocol.impl.CopyingInputStream.read(CopyingInputStream.java:39) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at java.io.FilterInputStream.read(FilterInputStream.java:83) ~[na:1.8.0_77]
 at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:109) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) [nifi-socket-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]


2017-01-18 22:09:59,610 WARN [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Failed to determine which node is elected active Cluster Coordinator: ZooKeeper reports the address as nifi-1.nifi.local:9088, but there is no node with this address. Attempted to determine the node's information but failed to retrieve its information due to org.apache.nifi.cluster.protocol.ProtocolException: Failed to request Node Identifer from nifi-1.nifi.local:9088

Also ranger is giving 409 errors when connecting to NiFi:

409 indicates a client issue (from Ranger). It seems I have some misconfiguration on NiFi now.

ydavis · ‎01-18-2017

Hi @Oliver Fletcher,

What configuration do you have for the ranger_nifi_plugin_properties? Also which logs did you see this error (Ranger or NiFi)?

View solution in original post

ydavis · ‎01-19-2017

Ok good progress so far! One thing that stands out is the Owner for Certificate (DN) used by Ranger. The nifi log posted appears to show that "CN=ranger-1, OU=Nifi, O=GR, L=London, ST=Unknown, C=Unknown" doesn't have access. I'm assuming that is the actual DN of the certificate used by Ranger. However in the ranger-nifi-plugin-properties section the Owner for Certificate value appears as "CN=ranger-1, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown". Nifi is expecting to identify and authorize Ranger by that value, however it doesn't appear that is the actual Owner info.

You should be able to update to the correct value using Ambari. So I suggest changing the owner.for.certificate in ranger-nifi-plugin-properties to match the actual value "CN=ranger-1, OU=Nifi, O=GR, L=London, ST=Unknown, C=Unknown" as described in Part 2, Step 3 i) on the community document. Just update that one field, save the configuration and restart NiFi. Behind the scenes the authorizers.xml configuration file for nifi should be updated with the values for Ranger Admin Identity. And that's what NiFi will use to identify when Ranger is attempting communication.

ydavis · ‎01-19-2017

Another thought on Solr. That actually lives behind the scenes of Ambari Infra. If you enabled auditing for the Ranger-NiFi plugin it should have populated configuration to use Solr that's behind Ambari Infra for logging (I believe it populates those values by default) . If you could post what you have configured for ranger-nifi-audit properties that would be easier for me to determine for sure.

ydavis · ‎01-19-2017

Lastly concerning the policies defined. If you could post a screen shot of what you have defined that would be helpful for me to troubleshoot as well.

ollie1 · ‎01-20-2017

More progress. I scripted up the creation of the truststore's and keystore's on both NiFi and Ranger so I was able to tear down and re-deploy the cluster consistently. I realised I'd made a few silly mistakes with the DN's you mentioned above. Fixing these gave me a 403 untrusted proxy, which I fixed by creating the /proxy policy for the nifi nodes.

I've now achieved:

Big step! And the policies are sync'ing with 200 OK's, as well as I can see active nifi user logging into Ranger. Seem's like I'm getting close. One issue left, is that my ldapsync in ranger has populated users & groups, but these users & groups when applied to the all resources policy don't appear to take effect. I have insufficient privileges to do anything in NiFi with a user I've granted access to inside Ranger:

For user oliver (oliver@NIFI,LOCAL), NiFi logs show a successful authentication, but unauthorised to access anything:

2017-01-20 14:43:11,282 INFO [NiFi Web Server-98] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for oliver@NIFI.LOCAL
2017-01-20 14:43:11,283 INFO [NiFi Web Server-98] o.a.n.w.a.c.AccessDeniedExceptionMapper oliver@NIFI.LOCAL does not have permission to access the requested resource. Returning Forbidden response.

I've setup NiFi using AD (ldaps) and Ranger using ldap (couldn't get ldaps to take). I'm not sure if that has triggered a weird issue here?

Thanks again for all your help!

ydavis · ‎01-20-2017

Hi @Oliver Fletcher! Great work making it this far. Ok here's the challenge. Unfortunately right now Ranger-NiFi plugin doesn't support groups in Ranger. This is a known issue and I believe there is work pending to address it. I see you do have a user entry of oliver, however is the username set to oliver@NIFI.LOCAL ? Based on your logs that is what NiFi is expecting to find.

ollie1 · ‎01-21-2017

Yes all working when I changed the attribute value in ranger LDAPS config to use UserPrincipalName, pulling in my users named ..@NIFI.LOCAL. Policy management is working as expected!

It's a shame that group permissions doesn't work yet, is there a work ticket I can follow its progress on?

ydavis · ‎01-23-2017

Glad that worked! Concerning group permission definitely a known issue, don't believe there's a public work ticket that you can follow.

Cloudera Community

Support Questions

Configuring Secured NiFi with Secured Ranger for Authorization Management