question Re: Configuring Secured NiFi with Secured Ranger for Authorization Management in Support Questions

Configuring Secured NiFi with Secured Ranger for Authorization Management

ollie1 — Mon, 19 Aug 2019 08:52:30 GMT

I'm having a torrid time trying to configure ranger with NiFi, with both services setup with SSL already. I've been following this guide: https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range.html

It was previously working without SSL, so something is wrong with my keystore's and truststores. I'll describe my setup in as much detail as possible, and I'm hoping, between my config, and logs, we can make some progress debugging the issue.

Overview:

Ambari (HDF-2.1.1.0) is managing a ranger and NIFI install, all on separate instances (ambari-1, nifi-1 and ranger-1). I have not configured a NIFI Certificate Authority.

NiFi instance:

Truststores:

/etc/security/nifi-certs/keystore.jks /etc/security/nifi-certs/truststore.jks

/etc/security/ranger-certs/keystore.jks /etc/security/ranger-certs/truststore.jks

nifi.properties

nifi.security.identity.mapping.pattern.dn=
nifi.security.identity.mapping.pattern.kerb=
nifi.security.identity.mapping.value.dn=
nifi.security.identity.mapping.value.kerb=
nifi.security.keyPasswd=easypass
nifi.security.keyPasswd.protected=aes/gcm/256
nifi.security.keystore=/etc/security/nifi-certs/keystore.jks
nifi.security.keystorePasswd=easypass
nifi.security.keystorePasswd.protected=aes/gcm/256
nifi.security.keystoreType=JKS
nifi.security.needClientAuth=False
nifi.security.ocsp.responder.certificate=
nifi.security.ocsp.responder.url=
nifi.security.truststore=/etc/security/nifi-certs/truststore.jks
nifi.security.truststorePasswd=easypass
nifi.security.truststorePasswd.protected=aes/gcm/256
nifi.security.truststoreType=JKS
nifi.security.user.authorizer=ranger-provider
nifi.security.user.login.identity.provider=kerberos-provider

ranger-policymgr-ssl.xml

 <configuration>
    <property>
      <name>owner.for.certificate</name>
      <value></value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.keystore</name>
      <value>/etc/security/nifi-certs/keystore.jks</value>
    </property>
   <property>
      <name>xasecure.policymgr.clientssl.keystore.credential.file</name>
      <value>jceks://file/etc/ranger/NiFi_nifi/cred.jceks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.keystore.password</name>
      <value>easypass</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore</name>
      <value>/etc/security/nifi-certs/truststore.jks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore.credential.file</name>
      <value>jceks://file/etc/ranger/NiFi_nifi/cred.jceks</value>
    </property>
    <property>
      <name>xasecure.policymgr.clientssl.truststore.password</name>
      <value>easypass</value>
    </property>
  </configuration>

No notable ERROR messages appearing in nifi-app.log

Ranger instance:

Truststores:

/etc/security/ranger-certs/keystore.jks /etc/security/ranger-certs/truststore.jks

ranger-admin-site.xml

   <property>
      <name>ranger.truststore.file</name>
      <value>/etc/security/ranger-certs/truststore.jks</value>
    </property>
    <property>
      <name>ranger.truststore.password</name>
      <value>easypass</value>
    </property>
    <property>
      <name>ranger.https.attrib.keystore.file</name>
      <value>/etc/security/ranger-certs/keystore.jks</value>
    </property>
  <property>
      <name>ranger.service.https.attrib.keystore.keyalias</name>
      <value>ranger-1</value>
    </property>
    <property>
      <name>ranger.service.https.attrib.keystore.pass</name>
      <value>easypass</value>
    </property>

Error logs (xa_portal.log) are showing that one of my keystore's password's is incorrect:

2017-01-18 19:40:54,646 [timed-executor-pool-0] ERROR org.apache.ranger.services.nifi.RangerServiceNiFi (RangerServiceNiFi.java:51) - <== RangerServiceNiFi.validateConfig Error:
java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.createSslContext(NiFiConnectionMgr.java:138)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.getNiFiClient(NiFiConnectionMgr.java:92)
        at org.apache.ranger.services.nifi.client.NiFiConnectionMgr.connectionTest(NiFiConnectionMgr.java:106)
        at org.apache.ranger.services.nifi.RangerServiceNiFi.validateConfig(RangerServiceNiFi.java:49)
        at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
        at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
        at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)

Though I've not been able to deduce which keystore this is complaining about!

and another REST ERROR

2017-01-18 20:03:45,901 [ranger-1.nifi.local-startStop-1] ERROR org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:138) - EmbeddedServiceDefsUtil.init(): failed
javax.ws.rs.WebApplicationException
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:311)
        at org.apache.ranger.service.RangerBaseModelService.read(RangerBaseModelService.java:234)
        at org.apache.ranger.biz.ServiceDBStore.getServiceDef(ServiceDBStore.java:1264)
        at org.apache.ranger.plugin.store.AbstractServiceStore.updateTagServiceDefForUpdatingAccessTypes(AbstractServiceStore.java:297)
        at org.apache.ranger.plugin.store.AbstractServiceStore.updateTagServiceDefForAccessTypes(AbstractServiceStore.java:55)
        at org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil.init(EmbeddedServiceDefsUtil.java:136)
        at org.apache.ranger.biz.ServiceDBStore$1.doInTransaction(ServiceDBStore.java:287)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at org.apache.ranger.biz.ServiceDBStore.initStore(ServiceDBStore.java:284)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Me

Thank's in advance for any help.

EDIT 1:

EDIT 2:

Step 1 & 2 in the guide:

[root@nifi-1 nifi-certs]# keytool -list -keystore truststore.jks
Enter keystore password:


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 2 entries


rootca, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 80:60:76:CF:8B:ED:37:79:73:3A:03:28:B3:9E:A9:AE:E9:03:EF:CD
mykey, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 9E:39:B3:8E:B3:37:76:2F:E5:99:CC:D1:13:E6:71:FC:1A:F1:C9:C8
[root@nifi-1 nifi-certs]#

Step 3 & 4:

[root@ranger-1 security]# cd /etc/security/ranger-certs/
[root@ranger-1 ranger-certs]# keytool -list -keystore truststore.jks
Enter keystore password:


Keystore type: JKS
Keystore provider: SUN


Your keystore contains 1 entry


nifi-1, 18-Jan-2017, trustedCertEntry,
Certificate fingerprint (SHA1): 9C:52:46:2D:90:3E:B7:24:D3:3F:0E:E4:21:DD:D6:0B:28:74:70:E4
[root@ranger-1 ranger-certs]#

EDIT 3:

Revised key and trust stores as @Yolanda M. Davis advised.

Errors above have stopped on the ranger node, and started on the NiFi.

2017-01-18 22:09:59,406 WARN [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-1.nifi.local due to javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_77]
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_77]
 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) ~[na:1.8.0_77]
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_77]
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:71) ~[na:1.8.0_77]
 at org.apache.nifi.cluster.protocol.impl.CopyingInputStream.read(CopyingInputStream.java:39) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at java.io.FilterInputStream.read(FilterInputStream.java:83) ~[na:1.8.0_77]
 at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:109) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:142) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) [nifi-socket-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]


2017-01-18 22:09:59,610 WARN [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Failed to determine which node is elected active Cluster Coordinator: ZooKeeper reports the address as nifi-1.nifi.local:9088, but there is no node with this address. Attempted to determine the node's information but failed to retrieve its information due to org.apache.nifi.cluster.protocol.ProtocolException: Failed to request Node Identifer from nifi-1.nifi.local:9088

Also ranger is giving 409 errors when connecting to NiFi:

409 indicates a client issue (from Ranger). It seems I have some misconfiguration on NiFi now.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 04:09:37 GMT

Hi @Oliver Fletcher,

What configuration do you have for the ranger_nifi_plugin_properties? Also which logs did you see this error (Ranger or NiFi)?

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 04:16:09 GMT

To add I'm concerned that the settings you need for Ranger to communicate securely with NiFi are not in place. Referring to https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range.html if you go to section 3, please confirm that you see the entries described in step 1 & 2. If not you can enter the information directly. Unfortunately Ranger doesn't currently allow us to update that setting through Ambari after it's initially created using Ambari.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ollie1 — Thu, 19 Jan 2017 04:20:02 GMT

Hi @yolanda

I've added a screen shot of ranger_nifi_plugin_properties, and the ERROR's are coming from ranger's xa_portal.log logfile.

I followed the steps in 1 & 2 - I'll do a keytool -list -v -keystore on the relevant stores, which should confirm they have been correctly exported. I'll add that as EDIT 2 to the post.

Thanks!

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 04:25:09 GMT

Another thing I'd suggest is to confirm that both the keystore/truststore that you've created for Ranger to use are accessible. I would manually run a keytool -list command:

e.g. keytool -list -v -keystore /etc/security/ranger-certs/keystore.jks

using the password you used to create the files. I'd run it on both the truststore and the keystore to confirm they are configured properly.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ollie1 — Thu, 19 Jan 2017 04:43:12 GMT

I've added those outputs. I'm going to empty my NiFi truststore and reimport the ranger certificate as I'm not sure why I have two certs in that store. Also I'll give it a more useful alias.

One question I have about your guide when setting up ranger-nifi-plugin-properties. This configures the ranger plugin sitting on the NiFi host right? The trust and key store that they need access to, are these the rangers trust and key store that need to be copied from the ranger host and distributed out to NiFi when the plugin is active? Maybe I've misunderstood that part..

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 04:46:57 GMT

I think you did it for keystores created for nifi but just wanted to check that both the keystore password and the key passwords are the same value for the key/truststores created for ranger?

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 05:14:51 GMT

The ranger-nifi-plugin-properties is actually used to configure the NiFi service repository in Ranger (the snapshot shown in Part 3 Step 2.) . Those settings help Ranger to be able to reach a secured NiFi in order to look up available rest endpoints that can be secured. When users initially enable the plugin in Ambari, update those values and choose to restart NiFi, Ambari will actually create the service repo populated with those values. The current challenge is when the Ranger plugin is enabled first without SSL settings. If a user goes back to add settings for SSL via Ambari unfortunately the api in Ranger doesn't support update of those fields by Ambari (which is why I suggested checking those settings directly in Ranger). I believe this is a known issue that has been logged (I'll confirm though).

The ranger-nifi-policymgr-ssl contains the settings that lives on the NiFi host (in a java credential file) which NiFi uses to talk to Ranger in order to retrieve policies that were configured and store them in it's local cache. Usually any issues with NiFi attempting to communicate with Ranger appear in the nifi-app.log. Also in Ranger you'll be able to see if the particular node connected or not from the Audit/Plugin tab.

I hope this makes sense. I'll review the document as well to see if I can make this a bit clearer.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 06:21:15 GMT

Going back to one of your early responses I think you said you saw two entries in your nifi truststore? I don't think you needed to clear those out; having two entries, that aren't duplicates shouldn't be a problem. The first entry may have been for that specific node or root CA if you used toolkit (I will need to research to check). The second would be your ranger cert.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ollie1 — Mon, 19 Aug 2019 08:52:17 GMT

Yes I didn't need to delete anything from the stores so I reverted that change. I think there was some issues with the key/trust stores which have been fixed, definitely making progress.

I now get an explicit 403 from ranger Service Manager > Edit Service:

and nifi-user.log shows:

2017-01-19 09:11:01,627 INFO [NiFi Web Server-16] o.a.n.w.a.c.AccessDeniedExceptionMapper CN=ranger-1, OU=Nifi, O=GR, L=London, ST=Unknown, C=Unknown does not have permission to access the requested resource. Returning Forbidden response.

Additionally, in Ranger -> Audit -> Plugins I can see that policies are being sync'd to NiFi:

As well as login attempts from NiFi to Ranger being registered:

However, nothing is being shown in Audit > Access, and I still receive an error message saying that I cannot connect to Audit Store. Ranger xa_portal.log also shows a big REST error every time I venture to that tab:

2017-01-19 09:19:03,676 [http-bio-6182-exec-4] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:336) - Operation error. response=VXResponse={org.apache.ranger.view.VXResponse@497448d5statusCode={1} msgDesc={Error connecting to search engine} messageList={[VXMessage={org.apache.ranger.view.VXMessage@1912a8acname={ERROR_SYSTEM} rbKey={xa.error.system} message={System Error. Please try later.} objectId={null} fieldName={null} }]} }
javax.ws.rs.WebApplicationException
 at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
 at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:335)
 at org.apache.ranger.solr.SolrAccessAuditsService.searchXAccessAudits(SolrAccessAuditsService.java:130)
 at org.apache.ranger.biz.AssetMgr.getAccessL

Finally, my access control policies defined in ranger for NiFi do not take effect, I have granted an AD domain account root access to NiFi, NiFi allows me to login in but tells me I have no privileges.

EDIT 1:

Maybe the 'Access' error is a red herring - I haven't installed any services other than ranger, nifi, zookeeper and kerberos in the cluster. So a connection to Solr doesn't make sense?

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 22:26:37 GMT

Ok good progress so far! One thing that stands out is the Owner for Certificate (DN) used by Ranger. The nifi log posted appears to show that "CN=ranger-1, OU=Nifi, O=GR, L=London, ST=Unknown, C=Unknown" doesn't have access. I'm assuming that is the actual DN of the certificate used by Ranger. However in the ranger-nifi-plugin-properties section the Owner for Certificate value appears as "CN=ranger-1, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown". Nifi is expecting to identify and authorize Ranger by that value, however it doesn't appear that is the actual Owner info.

You should be able to update to the correct value using Ambari. So I suggest changing the owner.for.certificate in ranger-nifi-plugin-properties to match the actual value "CN=ranger-1, OU=Nifi, O=GR, L=London, ST=Unknown, C=Unknown" as described in Part 2, Step 3 i) on the community document. Just update that one field, save the configuration and restart NiFi. Behind the scenes the authorizers.xml configuration file for nifi should be updated with the values for Ranger Admin Identity. And that's what NiFi will use to identify when Ranger is attempting communication.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 22:45:27 GMT

Another thought on Solr. That actually lives behind the scenes of Ambari Infra. If you enabled auditing for the Ranger-NiFi plugin it should have populated configuration to use Solr that's behind Ambari Infra for logging (I believe it populates those values by default) . If you could post what you have configured for ranger-nifi-audit properties that would be easier for me to determine for sure.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Thu, 19 Jan 2017 22:47:05 GMT

Lastly concerning the policies defined. If you could post a screen shot of what you have defined that would be helpful for me to troubleshoot as well.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ollie1 — Mon, 19 Aug 2019 08:51:57 GMT

More progress. I scripted up the creation of the truststore's and keystore's on both NiFi and Ranger so I was able to tear down and re-deploy the cluster consistently. I realised I'd made a few silly mistakes with the DN's you mentioned above. Fixing these gave me a 403 untrusted proxy, which I fixed by creating the /proxy policy for the nifi nodes.

I've now achieved:

Big step! And the policies are sync'ing with 200 OK's, as well as I can see active nifi user logging into Ranger. Seem's like I'm getting close. One issue left, is that my ldapsync in ranger has populated users & groups, but these users & groups when applied to the all resources policy don't appear to take effect. I have insufficient privileges to do anything in NiFi with a user I've granted access to inside Ranger:

For user oliver (oliver@NIFI,LOCAL), NiFi logs show a successful authentication, but unauthorised to access anything:

2017-01-20 14:43:11,282 INFO [NiFi Web Server-98] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for oliver@NIFI.LOCAL
2017-01-20 14:43:11,283 INFO [NiFi Web Server-98] o.a.n.w.a.c.AccessDeniedExceptionMapper oliver@NIFI.LOCAL does not have permission to access the requested resource. Returning Forbidden response.

I've setup NiFi using AD (ldaps) and Ranger using ldap (couldn't get ldaps to take). I'm not sure if that has triggered a weird issue here?

Thanks again for all your help!

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Sat, 21 Jan 2017 03:33:53 GMT

Hi @Oliver Fletcher! Great work making it this far. Ok here's the challenge. Unfortunately right now Ranger-NiFi plugin doesn't support groups in Ranger. This is a known issue and I believe there is work pending to address it. I see you do have a user entry of oliver, however is the username set to oliver@NIFI.LOCAL ? Based on your logs that is what NiFi is expecting to find.

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ollie1 — Sat, 21 Jan 2017 18:46:47 GMT

Yes all working when I changed the attribute value in ranger LDAPS config to use UserPrincipalName, pulling in my users named ..@NIFI.LOCAL. Policy management is working as expected!

It's a shame that group permissions doesn't work yet, is there a work ticket I can follow its progress on?

Re: Configuring Secured NiFi with Secured Ranger for Authorization Management

ydavis — Tue, 24 Jan 2017 00:51:59 GMT

Glad that worked! Concerning group permission definitely a known issue, don't believe there's a public work ticket that you can follow.