Skip to main content
myuintelli2021
myuintelli2021
Silver ⭐️⭐️⭐️
June 7, 2021
Question

Nifi untrusted proxy caused by Untrusted Proxy Exception thrown by X509AuthenticationProvider

I have setup a 3-node nifi cluster (i.e. nifi2, nifi3, and nifi4) using version 1.13.2 and I have enabled security and TLS using nifi tool kit.

 

Also, I have configured the nifi cluster to authenticate using openid connect against azure ad.

 

During the login, the browser was redirected to the azure portal for authentication, after successful login, it was redirected back to the nifi node but got "Untrusted proxy error" with following stack trace on every node:
...
2021-06-04 17:49:09,091 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2021-06-04 17:49:09,102 DEBUG [NiFi Web Server-17] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2021-06-04 17:49:09,102 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2021-06-04 17:49:09,107 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7)
2021-06-04 17:49:09,210 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user@{valid_domain}.com
2021-06-04 17:49:09,211 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: user@{valid_domain}.com
2021-06-04 17:49:09,211 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: user@{valid_domain}.com
2021-06-04 17:49:09,211 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: user@{valid_domain}.com
2021-06-04 17:49:09,891 DEBUG [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2021-06-04 17:49:09,891 DEBUG [NiFi Web Server-15] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <user@{valid_domain}.com>
2021-06-04 17:49:09,891 DEBUG [NiFi Web Server-15] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2021-06-04 17:49:09,892 INFO [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user@{valid_domain}.com><CN=nifi4.{valid_domain}.com, OU=NIFI>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7)
2021-06-04 17:49:09,896 WARN [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=nifi4.{valid_domain}.com, OU=NIFI
2021-06-04 17:49:09,897 DEBUG [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter
org.apache.nifi.web.security.UntrustedProxyException: Untrusted proxy CN=nifi4.{valid_domain}.com, OU=NIFI
at org.apache.nifi.web.security.x509.X509AuthenticationProvider.authenticate(X509AuthenticationProvider.java:133)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:79)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.lang.Thread.run(Unknown Source)

...

According to the log, the authentication was successful and the node, where authentication happened, tried to broadcast the authentication info to all nodes in the cluster.

But due to an unknown x509 related reason, the proxies (in listening mode) rejected the broadcasted authentication info.

I am aware there is a separate post @ https://community.cloudera.com/t5/Support-Questions/NiFi-Untrusted-proxy/m-p/150447/highlight/false#M112949; I tried the suggestions and but no luck.

What/what certificate might be causing org.apache.nifi.web.security.x509.X509AuthenticationProvider.authenticate(X509AuthenticationProvider.java:133) to fail? How to resolve it?

Many thanks to the help.

6 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
June 7, 2021

@myuintelli2021 

 

The "Untrusted Proxy Exception" issue has nothing to do with Authentication. It is an authorization issue within NiFi.

In a NiFi cluster, a users authenticates access to the specific node for which the opened the UI.  Subsequent request to access specific resource endpoints (like being able to view the UI), need to be  the sent to the cluster coordinator node.  The cluster coordinator then replicates that request to all nodes in the cluster and confirms each node responded to that request.

When for example node 1 needs to send a request made my user 1 to node 2, node 1 makes this request on behalf of the authenticated user 1.   So node 1 is proxying users 1's request.  So within NiFi authorizations, node 1 must be authorized to act as a proxy.

NiFi offers multiple authorizers, so the first question here is which authorizer are you using?

How is your authorizers.xml configured?

A very simple setup that utilizes a local managed authorizer might look like this:

 

<authorizers>

    <userGroupProvider>
        <identifier>file-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
        <property name="Users File">./conf/users.xml</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Initial User Identity 1">user@{valid_domain}.com</property>
        property name="Initial User Identity 2">CN=nifi2.{valid_domain}.com, OU=NIFI</property>
        property name="Initial User Identity 3">CN=nifi3.{valid_domain}.com, OU=NIFI</property>
        property name="Initial User Identity 4">CN=nifi4.{valid_domain}.com, OU=NIFI</property>
    </userGroupProvider>

    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">file-user-group-provider</property>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Initial Admin Identity">user@{valid_domain}.com</property>
        <property name="Legacy Authorized Users File"></property>
        <property name="Node Identity 1">CN=nifi2.{valid_domain}.com, OU=NIFI</property>
        <property name="Node Identity 1">CN=nifi3.{valid_domain}.com, OU=NIFI</property>
        <property name="Node Identity 1">CN=nifi4.{valid_domain}.com, OU=NIFI</property>
    </accessPolicyProvider>

    <authorizer>
        <identifier>managed-authorizer</identifier>
        <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
        <property name="Access Policy Provider">file-access-policy-provider</property>
    </authorizer>

</authorizers>

 

The "file-user-group-provider" is used here to generate the "users.xml" file initially.  Once created, this provider will not edit it if you make changes (only does anything if the file does NOT exist already).  So I would suggest you check this file to make sure yoru initial admin and all 3 of your nodes are present in this file (case sensitive).

The "file-access-policy-provider" is used here to generate the "authorizations.xml", which sets up the minimum necessary authorization policies needed fro your initial admin user and your NiFi cluster nodes.  If you look in this file, you will see numerous policies that your initial admins assigned UUID should be authorized for.  You should also see /proxy policies set in here with the assigned UUIDs from your NiFi nodes.

If these are missing, you should correct your authorizers.xml, remove the existing users.xml and authorizations.xml files, and restart your NiFi so these two files are created again.

Also noticed from your log snippet that it is identifying your nodes via their entire DN (CN=nifi2.{valid_domain}.com, OU=NIFI).   There exists the ability to setup identity mapping properties in the nifi.properties file which are used to reformat authenticated users/nodes.

For example, setting below three properties would change "CN=nifi2.{valid_domain}.com, OU=NIFI" into "nifi2.{valid_domain}.com":

nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=LOWER

Then all you would need to do is have the lowercase "nifi2.{valid_domain}.com" hostnames populated in your authorizers.xml file.


You can create as many sets of identity mapping properties as you like as long as each set using a unique string in the property names (above uses "dn"), but you could create another set using dn2, dn3, kerb2, kerb3, username, etc.   Just keep in mind that all authenticated user/client strings will be evaluated by these mapping properties in alphanumeric order and first matching pattern will have its value and transform applied. After that any additional pattern properties would not get evaluated.  

If you found this assisted with your query, please take a moment to login and click "accept" on this solution.
Thank you,
Matt

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
June 7, 2021

@myuintelli2021 

Noticed in another post from you that commented:

I am aware that there are 3 TLS certificates (one for each server) stored in keystore and 1 self-signed CA (stored in truststore) for nifi cluster.

 
NiFi keystore used in each node MUST meet following minimum criteria:
- Must contain ONLY 1 PrivateKeyEntry. Having more than 1 PrivateKeyEntry will not work as NiFi will not know which to use. 
- The DN used in the PrivateKeyEntry must not contain wildcards. Since NiFi certificate is used for ClientAuth, the PrivateKeyEntry DN is what is presented to identify the node.  Many Authorizers will not support client names with wildcards, plus it is not advisable security wise.
- The PrivateKeyEntry must have an Extended Key Usage (EKU) that supports both clientAuth and serverAuth
- The PrivateKeyEntry must have at least 1 SAN entry that matches the hostname for the server on which the keystore is being used.


Assuming since you used the NiFi CA toolkit to build your keystores and truststore files, you are good here. Just adding this detail in case you switch a some point to using private or publicly signed certificates.

Thanks,

Matt

Translate
myuintelli2021
myuintelli2021Author
Silver ⭐️⭐️⭐️
June 8, 2021

@MattWho ,

 

Thanks for the detailed explanation and suggestion.

 

Here is my modified authorizer.xml file:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<authorizers>


<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">user@{valid_domain}.com</property>

<property name="Initial User Identity 2">CN=nifi2.{valid_domain}.com</property>
<property name="Initial User Identity 3">CN=nifi3.{valid_domain}.com</property>
<property name="Initial User Identity 4">CN=nifi4.{valid_domain}.com</property>
</userGroupProvider>

<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">user@{valid_domain}.com</property>
<property name="Legacy Authorized Users File"></property>

<property name="Node Identity 1">CN=nifi2.{valid_domain}.com</property>
<property name="Node Identity 2">CN=nifi3.{valid_domain}.com</property>
<property name="Node Identity 3">CN=nifi4.{valid_domain}.com</property>
<property name="Node Group"></property>
</accessPolicyProvider>

<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>

</authorizers>

 

I added following lines to nifi.properties:
nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=LOWER

 

I also verified that keystore on each node only contains the certificate for that pariticular node like follows:

C:\nifi-1.13.2\conf>keytool -list --keystore ./keystore.jks -v
Enter keystore password:
Keystore type: jks
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: nifi-key
Creation date: Jun 2, 2021
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=nifi4.{valid_domain}.com, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 179cd17233f00000000
Valid from: Wed Jun 02 14:18:36 UTC 2021 until: Tue Sep 05 14:18:36 UTC 2023
Certificate fingerprints:
SHA1: 16:CC:2F:B0:A1:51:23:AD:8B:25:3D:EB:E4:C8:CF:89:49:20:C2:91
SHA256: 23:2A:9C:92:08:99:32:16:8A:AB:5F:DD:A3:A7:CC:C5:F7:B0:13:01:02:50:90:8B:35:50:D7:6D:BD:D5:38:E9
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3


But I still get the same error (i.e. unauthorized proxy after successful authentication):

2021-06-08 15:31:15,852 INFO [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7)
2021-06-08 15:31:15,859 INFO [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user@{valid_domain}.com
2021-06-08 15:31:15,981 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user@{valid_domain}.com><CN=nifi4.{valid_domain}.com, OU=NIFI>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7)
2021-06-08 15:31:15,985 WARN [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy nifi4.{valid_domain}.com
2021-06-08 15:33:19,173 INFO [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user@{valid_domain}.com><CN=nifi3.{valid_domain}.com, OU=NIFI>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.5)
2021-06-08 15:33:19,173 WARN [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy nifi3.{valid_domain}.com

 

What else might be causing this?

Many thanks.

Translate
Terms & ConditionsAccessibility statement