Configuring Secured NiFi with Secured Ranger for Authorization Management

[#1]

I'm having a torrid time trying to configure ranger with NiFi, with both services setup with SSL already. I've been following this guide: https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range....

It was previously working without SSL, so something is wrong with my keystore's and truststores. I'll describe my setup in as much detail as possible, and I'm hoping, between my config, and logs, we can make some progress debugging the issue.

Overview:

Ambari (HDF-2.1.1.0) is managing a ranger and NIFI install, all on separate instances (ambari-1, nifi-1 and ranger-1). I have not configured a NIFI Certificate Authority.

NiFi instance:

Truststores:

/etc/security/nifi-certs/keystore.jks /etc/security/nifi-certs/truststore.jks

/etc/security/ranger-certs/keystore.jks /etc/security/ranger-certs/truststore.jks

nifi.properties

nifi.security.identity.mapping.pattern.dn=
nifi.security.identity.mapping.pattern.kerb=
nifi.security.identity.mapping.value.dn=
nifi.security.identity.mapping.value.kerb=
nifi.security.keyPasswd=easypass
nifi.security.keyPasswd.protected=aes/gcm/256
nifi.security.keystore=/etc/security/nifi-certs/keystore.jks
nifi.security.keystorePasswd=easypass
nifi.security.keystorePasswd.protected=aes/gcm/256
nifi.security.keystoreType=JKS
nifi.security.needClientAuth=False
nifi.security.ocsp.responder.certificate=
nifi.security.ocsp.responder.url=
nifi.security.truststore=/etc/security/nifi-certs/truststore.jks
nifi.security.truststorePasswd=easypass
nifi.security.truststorePasswd.protected=aes/gcm/256
nifi.security.truststoreType=JKS
nifi.security.user.authorizer=ranger-provider
nifi.security.user.login.identity.provider=kerberos-provider

ranger-policymgr-ssl.xml

 <configuration>
    <property>
      <name>owner.for.certificate</name>
      <value></value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.keystore</name>
      <value>/etc/security/nifi-certs/keystore.jks</value>
    </property>
   <property>
      <name>xasecure.policymgr.clientssl.keystore.credential.file</name>
      <value>jceks://file/etc/ranger/NiFi_nifi/cred.jceks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.keystore.password</name>
      <value>easypass</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore</name>
      <value>/etc/security/nifi-certs/truststore.jks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore.credential.file</name>
      <value>jceks://file/etc/ranger/NiFi_nifi/cred.jceks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore.password</name>
      <value>easypass</value>
    </property>
  </configuration>

No notable ERROR messages appearing in nifi-app.log

Ranger instance:

Truststores:

/etc/security/ranger-certs/keystore.jks /etc/security/ranger-certs/truststore.jks

ranger-admin-site.xml

   <property>
      <name>ranger.truststore.file</name>
      <value>/etc/security/ranger-certs/truststore.jks</value>
    </property>
    <property>
      <name>ranger.truststore.password</name>
      <value>easypass</value>
    </property>
    <property>
      <name>ranger.https.attrib.keystore.file</name>
      <value>/etc/security/ranger-certs/keystore.jks</value>
    </property>
  <property>
      <name>ranger.service.https.attrib.keystore.keyalias</name>
      <value>ranger-1</value>
    </property>
    <property>
      <name>ranger.service.https.attrib.keystore.pass</name>
      <value>easypass</value>
    </property>

Error logs (xa_portal.log) are showing that one of my keystore's password's is incorrect:

2017-01-18 19:40:54,646 [timed-executor-pool-0] ERROR org.apache.ranger.services.nifi.RangerServiceNiFi (RangerServiceNiFi.java:51) - <== RangerServiceNiFi.validateConfig Error:
java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.createSslContext(NiFiConnectionMgr.java:138)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.getNiFiClient(NiFiConnectionMgr.java:92)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.connectionTest(NiFiConnectionMgr.java:106)
        at org.apache.ranger.services.nifi.RangerServiceNiFi.validateConfig(RangerServiceNiFi.java:49)
        at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
        at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
        at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)

Though I've not been able to deduce which keystore this is complaining about!

and another REST ERROR

2017-01-18 20:03:45,901 [ranger-1.nifi.local-startStop-1] ERROR org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:138) - EmbeddedServiceDefsUtil.init(): failed
javax.ws.rs.WebApplicationException
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:311)
        at org.apache.ranger.service.RangerBaseModelService.read(RangerBaseModelService.java:234)
        at org.apache.ranger.biz.ServiceDBStore.getServiceDef(ServiceDBStore.java:1264)
        at org.apache.ranger.plugin.store.AbstractServiceStore.updateTagServiceDefForUpdatingAccessTypes(AbstractServiceStore.java:297)
        at org.apache.ranger.plugin.store.AbstractServiceStore.updateTagServiceDefForAccessTypes(AbstractServiceStore.java:55)
        at org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil.init(EmbeddedServiceDefsUtil.java:136)
        at org.apache.ranger.biz.ServiceDBStore$1.doInTransaction(ServiceDBStore.java:287)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at org.apache.ranger.biz.ServiceDBStore.initStore(ServiceDBStore.java:284)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Me

Thank's in advance for any help.

EDIT 1:

EDIT 2:

Step 1 & 2 in the guide:

[root@nifi-1 nifi-certs]# keytool -list -keystore truststore.jks
Enter keystore password:


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 2 entries


rootca, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 80:60:76:CF:8B:ED:37:79:73:3A:03:28:B3:9E:A9:AE:E9:03:EF:CD
mykey, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 9E:39:B3:8E:B3:37:76:2F:E5:99:CC:D1:13:E6:71:FC:1A:F1:C9:C8
[root@nifi-1 nifi-certs]#

Step 3 & 4:

[root@ranger-1 security]# cd /etc/security/ranger-certs/
[root@ranger-1 ranger-certs]# keytool -list -keystore truststore.jks
Enter keystore password:


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 1 entry


nifi-1, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 9C:52:46:2D:90:3E:B7:24:D3:3F:0E:E4:21:DD:D6:0B:28:74:70:E4
[root@ranger-1 ranger-certs]#

EDIT 3:

Revised key and trust stores as @Yolanda M. Davis advised.

Errors above have stopped on the ranger node, and started on the NiFi.

2017-01-18 22:09:59,406 WARN [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-1.nifi.local due to javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_77]
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) ~[na:1.8.0_77]
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_77]
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:71) ~[na:1.8.0_77]
 at org.apache.nifi.cluster.protocol.impl.CopyingInputStream.read(CopyingInputStream.java:39) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at java.io.FilterInputStream.read(FilterInputStream.java:83) ~[na:1.8.0_77]
 at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:109) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) [nifi-socket-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]


2017-01-18 22:09:59,610 WARN [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Failed to determine which node is elected active Cluster Coordinator: ZooKeeper reports the address as nifi-1.nifi.local:9088, but there is no node with this address. Attempted to determine the node's information but failed to retrieve its information due to org.apache.nifi.cluster.protocol.ProtocolException: Failed to request Node Identifer from nifi-1.nifi.local:9088

Also ranger is giving 409 errors when connecting to NiFi:

409 indicates a client issue (from Ranger). It seems I have some misconfiguration on NiFi now.

[#2 · Accepted solution]

Hi @Oliver Fletcher,

What configuration do you have for the ranger_nifi_plugin_properties? Also which logs did you see this error (Ranger or NiFi)?

[#3]

To add I'm concerned that the settings you need for Ranger to communicate securely with NiFi are not in place. Referring to https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range.... if you go to section 3, please confirm that you see the entries described in step 1 & 2. If not you can enter the information directly. Unfortunately Ranger doesn't currently allow us to update that setting through Ambari after it's initially created using Ambari.

[#4]

Another thing I'd suggest is to confirm that both the keystore/truststore that you've created for Ranger to use are accessible. I would manually run a keytool -list command:

e.g. keytool -list -v -keystore /etc/security/ranger-certs/keystore.jks

using the password you used to create the files. I'd run it on both the truststore and the keystore to confirm they are configured properly.

[#5]

I've added those outputs. I'm going to empty my NiFi truststore and reimport the ranger certificate as I'm not sure why I have two certs in that store. Also I'll give it a more useful alias.

One question I have about your guide when setting up ranger-nifi-plugin-properties. This configures the ranger plugin sitting on the NiFi host right? The trust and key store that they need access to, are these the rangers trust and key store that need to be copied from the ranger host and distributed out to NiFi when the plugin is active? Maybe I've misunderstood that part..

[#6]

The ranger-nifi-plugin-properties is actually used to configure the NiFi service repository in Ranger (the snapshot shown in Part 3 Step 2.) . Those settings help Ranger to be able to reach a secured NiFi in order to look up available rest endpoints that can be secured. When users initially enable the plugin in Ambari, update those values and choose to restart NiFi, Ambari will actually create the service repo populated with those values. The current challenge is when the Ranger plugin is enabled first without SSL settings. If a user goes back to add settings for SSL via Ambari unfortunately the api in Ranger doesn't support update of those fields by Ambari (which is why I suggested checking those settings directly in Ranger). I believe this is a known issue that has been logged (I'll confirm though).

The ranger-nifi-policymgr-ssl contains the settings that lives on the NiFi host (in a java credential file) which NiFi uses to talk to Ranger in order to retrieve policies that were configured and store them in it's local cache. Usually any issues with NiFi attempting to communicate with Ranger appear in the nifi-app.log. Also in Ranger you'll be able to see if the particular node connected or not from the Audit/Plugin tab.

I hope this makes sense. I'll review the document as well to see if I can make this a bit clearer.

[#7]

Hi @yolanda

I've added a screen shot of ranger_nifi_plugin_properties, and the ERROR's are coming from ranger's xa_portal.log logfile.

I followed the steps in 1 & 2 - I'll do a keytool -list -v -keystore on the relevant stores, which should confirm they have been correctly exported. I'll add that as EDIT 2 to the post.

Thanks!

[#8]

I think you did it for keystores created for nifi but just wanted to check that both the keystore password and the key passwords are the same value for the key/truststores created for ranger?

[#9]

Going back to one of your early responses I think you said you saw two entries in your nifi truststore? I don't think you needed to clear those out; having two entries, that aren't duplicates shouldn't be a problem. The first entry may have been for that specific node or root CA if you used toolkit (I will need to research to check). The second would be your ranger cert.

[#10]

Yes I didn't need to delete anything from the stores so I reverted that change. I think there was some issues with the key/trust stores which have been fixed, definitely making progress.

I now get an explicit 403 from ranger Service Manager > Edit Service:

and nifi-user.log shows:

2017-01-19 09:11:01,627 INFO [NiFi Web Server-16] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=ranger-1, OU=Nifi, O=GR, L=London, ST=Unknown, C=Unknown does not have permission to access the requested resource. Returning Forbidden response.

Additionally, in Ranger -> Audit -> Plugins I can see that policies are being sync'd to NiFi:

As well as login attempts from NiFi to Ranger being registered:

However, nothing is being shown in Audit > Access, and I still receive an error message saying that I cannot connect to Audit Store. Ranger xa_portal.log also shows a big REST error every time I venture to that tab:

2017-01-19 09:19:03,676 [http-bio-6182-exec-4] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:336) - Operation error. response=VXResponse={org.apache.ranger.view.VXResponse@497448d5statusCode={1} msgDesc={Error connecting to search engine} messageList={[VXMessage={org.apache.ranger.view.VXMessage@1912a8acname={ERROR_SYSTEM} rbKey={xa.error.system} message={System Error. Please try later.} objectId={null} fieldName={null} }]} }
javax.ws.rs.WebApplicationException
 at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
 at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:335)
 at org.apache.ranger.solr.SolrAccessAuditsService.searchXAccessAudits(SolrAccessAuditsService.java:130)
 at org.apache.ranger.biz.AssetMgr.getAccessL

Finally, my access control policies defined in ranger for NiFi do not take effect, I have granted an AD domain account root access to NiFi, NiFi allows me to login in but tells me I have no privileges.

EDIT 1:

Maybe the 'Access' error is a red herring - I haven't installed any services other than ranger, nifi, zookeeper and kerberos in the cluster. So a connection to Solr doesn't make sense?