Member since
05-26-2021
14
Posts
1
Kudos Received
0
Solutions
01-23-2023
12:42 PM
Hi! Matt, Thanks for your follow up. Our nifi 1.15.3 runs on windows 2019 datacenter edition and JDK version is 1.8. We're aware that JDK version might be old. Do you think we should upgrade? Thanks, Ming
... View more
01-18-2023
08:10 AM
Hi! Matt, Thanks for the help! I tried your logback.xml configuration in our env and it did not make difference. I noticed that the nifi version you used for testing is 1.19.1 while we use older version 1.15.3. Do you know if the log rotation feature was implemented in v1.15.3 or only available in a later version?
... View more
01-16-2023
07:49 AM
Hi! Folks, For some reason, it seems that the nifi service does not take the logging configurations of logback.xml and the service still generates the logs whose size exceed 100MB. We simply use the default logback.xml settings such as follow: <configuration scan="true" scanPeriod="30 seconds"> <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"> <resetJUL>true</resetJUL> </contextListener> <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>10</maxHistory> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> ... The file resides in the nifi service's conf folder "C:\nifi-1.15.3\conf\logback.xml". According to the config, log "c:\nifi-1.15.3\logs\nifi-app.log" should rotate when its size reaches 100MB. However, for some reason, the log rotation never happens. I did some some research and it doesn't seem config file "logback.xml" needs to be referenced or included in other nifi property file (e.g., c:\nifi-1.15.3\nifi.properties) to be effective. Please help.
... View more
03-28-2022
08:35 AM
By checking the status of https://issues.apache.org/jira/browse/NIFIREG-313, it seems that the feature of authenticating using OIDC for nifi registry is already resolved as of 9/14/2020. But In nifi registry admin guide at https://nifi.apache.org/docs/nifi-registry-docs/index.html, it still mentions that only LDAP and Kerberos are supported. Is the document up-to-date?
... View more
06-14-2021
11:56 AM
1 Kudo
@VidyaSargur We have not resolved the issue yet due to some priority shift. We're still seeing the same error. Will follow up on it soon. Ming
... View more
06-08-2021
08:52 AM
@MattWho , Thanks for the detailed explanation and suggestion. Here is my modified authorizer.xml file: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">user@{valid_domain}.com</property> <property name="Initial User Identity 2">CN=nifi2.{valid_domain}.com</property> <property name="Initial User Identity 3">CN=nifi3.{valid_domain}.com</property> <property name="Initial User Identity 4">CN=nifi4.{valid_domain}.com</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">user@{valid_domain}.com</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=nifi2.{valid_domain}.com</property> <property name="Node Identity 2">CN=nifi3.{valid_domain}.com</property> <property name="Node Identity 3">CN=nifi4.{valid_domain}.com</property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers> I added following lines to nifi.properties: nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$ nifi.security.identity.mapping.value.dn=$1 nifi.security.identity.mapping.transform.dn=LOWER I also verified that keystore on each node only contains the certificate for that pariticular node like follows: C:\nifi-1.13.2\conf>keytool -list --keystore ./keystore.jks -v Enter keystore password: Keystore type: jks Keystore provider: SUN Your keystore contains 1 entry Alias name: nifi-key Creation date: Jun 2, 2021 Entry type: PrivateKeyEntry Certificate chain length: 2 Certificate[1]: Owner: CN=nifi4.{valid_domain}.com, OU=NIFI Issuer: CN=localhost, OU=NIFI Serial number: 179cd17233f00000000 Valid from: Wed Jun 02 14:18:36 UTC 2021 until: Tue Sep 05 14:18:36 UTC 2023 Certificate fingerprints: SHA1: 16:CC:2F:B0:A1:51:23:AD:8B:25:3D:EB:E4:C8:CF:89:49:20:C2:91 SHA256: 23:2A:9C:92:08:99:32:16:8A:AB:5F:DD:A3:A7:CC:C5:F7:B0:13:01:02:50:90:8B:35:50:D7:6D:BD:D5:38:E9 Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 But I still get the same error (i.e. unauthorized proxy after successful authentication): 2021-06-08 15:31:15,852 INFO [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7) 2021-06-08 15:31:15,859 INFO [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user@{valid_domain}.com 2021-06-08 15:31:15,981 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user@{valid_domain}.com><CN=nifi4.{valid_domain}.com, OU=NIFI>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7) 2021-06-08 15:31:15,985 WARN [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy nifi4.{valid_domain}.com 2021-06-08 15:33:19,173 INFO [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user@{valid_domain}.com><CN=nifi3.{valid_domain}.com, OU=NIFI>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.5) 2021-06-08 15:33:19,173 WARN [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy nifi3.{valid_domain}.com What else might be causing this? Many thanks.
... View more
06-07-2021
06:44 AM
I have setup a 3-node nifi cluster (i.e. nifi2, nifi3, and nifi4) using version 1.13.2 and I have enabled security and TLS using nifi tool kit. Also, I have configured the nifi cluster to authenticate using openid connect against azure ad. During the login, the browser was redirected to the azure portal for authentication, after successful login, it was redirected back to the nifi node but got "Untrusted proxy error" with following stack trace on every node: ... 2021-06-04 17:49:09,091 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null 2021-06-04 17:49:09,102 DEBUG [NiFi Web Server-17] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request. 2021-06-04 17:49:09,102 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null 2021-06-04 17:49:09,107 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7) 2021-06-04 17:49:09,210 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user@{valid_domain}.com 2021-06-04 17:49:09,211 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: user@{valid_domain}.com 2021-06-04 17:49:09,211 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: user@{valid_domain}.com 2021-06-04 17:49:09,211 DEBUG [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: user@{valid_domain}.com 2021-06-04 17:49:09,891 DEBUG [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null 2021-06-04 17:49:09,891 DEBUG [NiFi Web Server-15] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <user@{valid_domain}.com> 2021-06-04 17:49:09,891 DEBUG [NiFi Web Server-15] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <> 2021-06-04 17:49:09,892 INFO [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user@{valid_domain}.com><CN=nifi4.{valid_domain}.com, OU=NIFI>) GET https://nifi4.{valid_domain}.com/nifi-api/flow/current-user (source ip: 10.2.2.7) 2021-06-04 17:49:09,896 WARN [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=nifi4.{valid_domain}.com, OU=NIFI 2021-06-04 17:49:09,897 DEBUG [NiFi Web Server-15] o.a.n.w.s.NiFiAuthenticationFilter org.apache.nifi.web.security.UntrustedProxyException: Untrusted proxy CN=nifi4.{valid_domain}.com, OU=NIFI at org.apache.nifi.web.security.x509.X509AuthenticationProvider.authenticate(X509AuthenticationProvider.java:133) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174) at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:79) at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) at java.lang.Thread.run(Unknown Source) ... According to the log, the authentication was successful and the node, where authentication happened, tried to broadcast the authentication info to all nodes in the cluster. But due to an unknown x509 related reason, the proxies (in listening mode) rejected the broadcasted authentication info. I am aware there is a separate post @ https://community.cloudera.com/t5/Support-Questions/NiFi-Untrusted-proxy/m-p/150447/highlight/false#M112949; I tried the suggestions and but no luck. What/what certificate might be causing org.apache.nifi.web.security.x509.X509AuthenticationProvider.authenticate(X509AuthenticationProvider.java:133) to fail? How to resolve it? Many thanks to the help.
... View more
Labels:
- Labels:
-
Apache NiFi