nifi-app.log:2020-06-01 21:46:41,079 WARN [Timer-Driven Process Thread-47] o.e.j.util.component.AbstractLifeCycle FAILED ServerConnector@1b1f0574{HTTP/1.1,[http/1.1]}{0.0.0.0:5151}: java.net.BindException: Address already in use nifi-app.log:java.net.BindException: Address already in use     i see this error from other nodes in the cluster. unless i update the processor to new port, we can run the flows  ... View more

exact same error with 1.9. enabled debugging and below is the log    https://issues.apache.org/jira/browse/NIFI-5522       2020-06-01 21:23:15,185 DEBUG [NiFi Web Server-466504] o.e.jetty.server.handler.ContextHandler scope null||/nifi-api/processors/bdac38da-b5cf-1e57-9f5e-ef744cf5204f/run-status @ o.e.j.w.WebAppContext@275bf9b3{nifi-docs,/nifi-docs,file:///grid/var/lib/nifi/work/jetty/nifi-web-docs-1.9.0.3.4.1.1-4.war/webapp/,AVAILABLE}{/grid/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.3.4.1.1-4.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-docs-1.9.0.3.4.1.1-4.war} 2020-06-01 21:23:15,185 DEBUG [NiFi Web Server-466504] o.e.jetty.server.handler.ContextHandler scope null||/nifi-api/processors/bdac38da-b5cf-1e57-9f5e-ef744cf5204f/run-status @ o.e.j.w.WebAppContext@27adc16e{nifi-error,/,file:///grid/var/lib/nifi/work/jetty/nifi-web-error-1.9.0.3.4.1.1-4.war/webapp/,AVAILABLE}{/grid/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.3.4.1.1-4.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.0.3.4.1.1-4.war} 2020-06-01 21:23:15,185 DEBUG [NiFi Web Server-466504] org.eclipse.jetty.server.Server handled=true async=false committed=true on HttpChannelOverHttp@4700a37d{r=30,c=true,a=DISPATCHED,uri=https://abc.com:9091/nifi-api/processors/bdac38da-b5cf-1e57-9f5e-ef744cf5204f/run-status,age=546} 2020-06-01 21:23:15,185 DEBUG [NiFi Web Server-466504] org.eclipse.jetty.server.HttpChannel HttpChannelOverHttp@4700a37d{r=30,c=true,a=COMPLETING,uri=https://abc.com:9091/nifi-api/processors/bdac38da-b5cf-1e57-9f5e-ef744cf5204f/run-status,age=546} action COMPLETE 2020-06-01 21:23:15,185 DEBUG [NiFi Web Server-466504] org.eclipse.jetty.server.HttpChannel COMPLETE for /nifi-api/processors/bdac38da-b5cf-1e57-9f5e-ef744cf5204f/run-status written=2861 Value: 'HandleHttpRequest[id=bdac38da-b5cf-1e57-9f5e-ef744cf5204f] Failed to process session due to Failed to initialize the server: org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server' Value: 'bdac38da-b5cf-1e57-9f5e-ef744cf5204f' Value: 'HandleHttpRequest[id=bdac38da-b5cf-1e57-9f5e-ef744cf5204f] Failed to process session due to Failed to initialize the server: org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server' Value: 'bdac38da-b5cf-1e57-9f5e-ef744cf5204f' Value: 'HandleHttpRequest[id=bdac38da-b5cf-1e57-9f5e-ef744cf5204f] Failed to process session due to Failed to initialize the server: org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server' Value: 'bdac38da-b5cf-1e57-9f5e-ef744cf5204f' Value: 'HandleHttpRequest[id=bdac38da-b5cf-1e57-9f5e-ef744cf5204f] Failed to process session due to Failed to initialize the server: org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server' Value: 'bdac38da-b5cf-1e57-9f5e-ef744cf5204f' Value: 'HandleHttpRequest[id=bdac38da-b5cf-1e57-9f5e-ef744cf5204f] Failed to process session due to Failed to initialize the server: org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server' Value: 'bdac38da-b5cf-1e57-9f5e-ef744cf5204f'   ... View more

@MattWho please check this   NIFI UI Invalid State home The Flow Controller is initializing the Data Flow.     meout 4000 maxSessionTimeout 40000 datadir ./state/zookeeper/version-2 snapdir ./state/zookeeper/version-2 2020-03-09 17:08:09,015 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2020-03-09 17:08:09,019 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-09 17:08:09,021 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)   2020-03-09 17:09:17,137 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-09 17:09:17,238 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-09 17:09:17,238 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-09 17:09:17,338 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-09 17:09:17,339 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-03-09 17:09:17,439 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up ... View more

@MattWho As per this document, have to generate certs for server and client @pvillard  https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/ whereas this shows more granular  one cert shared across all nodes in a cluster https://community.cloudera.com/t5/Community-Articles/Using-the-TLS-Toolkit-to-simplify-security/ta-p/247531   please recommend  ... View more

i have generated certs through toolkit    tls-toolkit.sh standalone -n 'localhost(3)' -C 'CN=Nifi_Admin,OU=NIFI' -O -o /home/ec2-user/   will this not work? <property name="Initial User Identity 1">CN=Nifi_Admin,OU=NIFI</property> <property name="Initial User Identity 2">CN=Nifi_Admin,OU=NIFI</property> <property name="Initial User Identity 3">CN=Nifi_Admin,OU=NIFI</property>     <property name="Node Identity 1">CN=Nifi_Admin,OU=NIFI</property> <property name="Node Identity 2">CN=Nifi_Admin,OU=NIFI</property> <property name="Node Identity 3">CN=Nifi_Admin,OU=NIFI</property> ... View more

here is the error I'm facing now, looks like AD access issue or firewall issues   Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.springframework.ldap.CommunicationException: ldap.ad.xx.xx:389; nested exception is javax.naming.CommunicationException: ldap.ad.xx.xx:389 [Root exception is java.net.SocketTimeoutException: connect timed out] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) ... View more

HI @MattWho i have configured authorizers.xml by referring this document available by @pvillard    could you check if the below updated conf file looks good?   authorizers.xml <authorizers>   <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">/data05/nifi/conf/users.xml</property> <property name="Legacy Authorized Users File" /> <property name="Initial User Identity 1">CN=Nifi_Admin,OU=NIFI</property> <property name="Initial User Identity 2">CN=Nifi_Admin,OU=NIFI</property> <property name="Initial User Identity 3">CN=Nifi_Admin,OU=NIFI</property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">CN=ap-unix-ldap,OU=UNIX Service Accounts,OU=Service Accounts,OU=Accounts,OU=New OU Structure,OU=AD Infrastructure,DC=AD,DC=XX,DC=XX</property> <property name="Manager Password">>XXXXX</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Page size"/> <property name="Sync Interval">20 min</property> <property name="Url">ldap://ldap.ad.XXX.edu:389</property> <property name="User Search Base">DC=AD,DC=XXX,DC=XX</property> <property name="User Object Class">user</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter">(|(memberOf=CN=ITS-UAP,OU=Official Groups,OU=ITS,DC=AD,DC=XXX,DC=XXX)(memberOf=CN=nifi-users,OU=CI-UBPS,OU=AD Infrastructure,DC=AD,DC=XXX,DC=XXX))</property> <property name="User Identity Attribute">sAMAccountName</property> <property name="User Group Name Attribute"/> <property name="User Group Name Attribute - Referenced Group Attribute"/> <property name="Group Search Base">DC=AD,DC=XX,DC=XX</property> <property name="Group Object Class">group</property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter">(|(CN=nifi-users)(CN=ITS-UAP))</property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"/> </userGroupProvider> <userGroupProvider> <identifier>composite-configurable-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">composite-configurable-user-group-provider</property> <property name="Authorizations File">/data05/nifi/conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=admin, OU=NIFI</property> <property name="Legacy Authorized Users File" />   <property name="Node Identity 1">CN=Nifi_Admin,OU=NIFI</property> <property name="Node Identity 2">CN=Nifi_Admin,OU=NIFI</property> <property name="Node Identity 3">CN=Nifi_Admin,OU=NIFI</property> </accessPolicyProvider> <authorizer> <identifier>ranger-provider</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers> ... View more

here is the log rotation and no errors but when i list for https port 9696 its not listening   [root@ip-172-31-35-116 conf]# netstat | grep -i 9696     te.internal:9696, state=CONNECTED, updateId=11] to NodeConnectionStatus[nodeId=ip-172-31-35-116.us-west-2.compute.internal:9696, state=CONNECTED, updateId=11] 2020-01-16 07:57:49,836 INFO [Process Cluster Protocol Request-13] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 798f4507-f2ab-47d3-bca9-3e64de45da5c (type=NODE_STATUS_CHANGE, length=1369 bytes) from ip-172-31-38-192.us-west-2.compute.internal in 97 millis 2020-01-16 07:57:49,837 INFO [Process Cluster Protocol Request-29] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 6d541bc2-26ba-4c8b-ae3b-679a8846d62c (type=NODE_STATUS_CHANGE, length=1369 bytes) from ip-172-31-38-192.us-west-2.compute.internal in 96 millis 2020-01-16 07:57:49,898 INFO [Process Cluster Protocol Request-21] o.a.n.c.c.node.NodeClusterCoordinator Status of ip-172-31-38-192.us-west-2.compute.internal:9696 changed from NodeConnectionStatus[nodeId=ip-172-31-38-192.us-west-2.compute.internal:9696, state=CONNECTED, updateId=12] to NodeConnectionStatus[nodeId=ip-172-31-38-192.us-west-2.compute.internal:9696, state=CONNECTED, updateId=12] 2020-01-16 07:57:49,899 INFO [Process Cluster Protocol Request-15] o.a.n.c.c.node.NodeClusterCoordinator Status of ip-172-31-38-192.us-west-2.compute.internal:9696 changed from NodeConnectionStatus[nodeId=ip-172-31-38-192.us-west-2.compute.internal:9696, state=CONNECTED, updateId=12] to NodeConnectionStatus[nodeId=ip-172-31-38-192.us-west-2.compute.internal:9696, state=CONNECTED, updateId=12] 2020-01-16 07:57:49,900 INFO [Process Cluster Protocol Request-17] o.a.n.c.c.node.NodeClusterCoordinator Status of ip-172-31-45-114.us-west-2.compute.internal:9696 changed from NodeConnectionStatus[nodeId=ip-172-31-45-114.us-west-2.compute.internal:9696, state=CONNECTED, updateId=10] to NodeConnectionStatus[nodeId=ip-172-31-45-114.us-west-2.compute.internal:9696, state=CONNECTED, updateId=10] 2020-01-16 07:57:49,902 INFO [Process Cluster Protocol Request-15] o.a.n.c.p.impl.SocketProtocolListener Finished processing request c82e760c-04f9-46cf-aee2-2676215b7d37 (type=NODE_STATUS_CHANGE, length=1369 bytes) from ip-172-31-38-192.us-west-2.compute.internal in 126 millis 2020-01-16 07:57:49,902 INFO [Process Cluster Protocol Request-17] o.a.n.c.p.impl.SocketProtocolListener Finished processing request ccc9a53b-0a5d-4621-98c5-825c4dfabc69 (type=NODE_STATUS_CHANGE, length=1369 bytes) from ip-172-31-38-192.us-west-2.compute.internal in 117 millis 2020-01-16 07:57:49,902 INFO [Process Cluster Protocol Request-21] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 23f5c858-a99b-434a-8a2d-63219e3ddf2c (type=NODE_STATUS_CHANGE, length=1369 bytes) from ip-172-31-38-192.us-west-2.compute.internal in 126 millis 2020-01-16 07:57:49,914 INFO [Process Cluster Protocol Request-14] o.a.n.c.c.node.NodeClusterCoordinator Status of ip-172-31-35-116.us-west-2.compute.internal:9696 changed from NodeConnectionStatus[nodeId=ip-172-31-35-116.us-west-2.compute.internal:9696, state=CONNECTED, updateId=11] to NodeConnectionStatus[nodeId=ip-172-31-35-116.us-west-2.compute.internal:9696, state=CONNECTED, updateId=11] 2020-01-16 07:57:49,925 INFO [Process Cluster Protocol Request-14] o.a.n.c.p.impl.SocketProtocolListener Finished processing request b828b7b5-1f44-4230-bd5a-d72b1f07f0da (type=NODE_STATUS_CHANGE, length=1369 bytes) from ip-172-31-38-192.us-west-2.compute.internal in 113 millis 2020-01-16 07:57:49,945 INFO [Process Cluster Protocol Request-6] o.a.n.c.c.node.NodeClusterCoordinator Status of ip-172-31-38-192.us-west-2.compute.internal:9696 changed from NodeConnectionStatus[nodeId=ip-172-31-38-192.us-west-2.compute.internal:9696, state=CONNECTED, updateId=12] to NodeConnectionStatus[nodeId=ip-172-31-38-192.us-west-2.compute.internal:9696, state=CONNECTED, updateId=12] 2020-01-16 07:57:49,952 INFO [Process Cluster Protocol Request-20] o.a.n.c.c.node.NodeClusterCoordinator Status of ip-172-31-45-114.us-west-2.compute.internal:9696 changed from NodeConnectionStatus[nodeId=ip-172-31-45-114.us-west-2.compute.internal:9696, state=CONNECTED, updateId=10] to NodeConnectionStatus[nodeId=ip-172-31-45-114.us-west-2.compute.internal:9696, state=CONNECTED, updateId=10] ^C [root@ip-172-31-35-116 conf]# netstat | grep -i 9696 ... View more

@pvillard @MattWho @TimothySpann @sunile_manjee  guys appreciate your inputs  nifi is running in the background u=but UI not coming up, don't see any error in the log I have enabled tls SSL certs on my three node cluster and enabled AD    here are my configs looks like  1) nifi.properties cat nifi.properties # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Core Properties # #nifi.flow.configuration.file=./conf/flow.xml.gz nifi.flow.configuration.file=/data05/nifi/conf/flow.xml.gz nifi.flow.configuration.archive.enabled=true #nifi.flow.configuration.archive.dir=./conf/archive/ nifi.flow.configuration.archive.dir=/data05/nifi/conf/archive/ nifi.flow.configuration.archive.max.time=30 days nifi.flow.configuration.archive.max.storage=500 MB nifi.flow.configuration.archive.max.count= nifi.flowcontroller.autoResumeState=true nifi.flowcontroller.graceful.shutdown.period=10 sec nifi.flowservice.writedelay.interval=500 ms nifi.administrative.yield.duration=30 sec # If a component has no work to do (is "bored"), how long should we wait before checking again for work? nifi.bored.yield.duration=10 millis nifi.queue.backpressure.count=10000 nifi.queue.backpressure.size=1 GB #nifi.authorizer.configuration.file=./conf/authorizers.xml #nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml #nifi.templates.directory=./conf/templates #nifi.authorizer.configuration.file=/data05/nifi/conf/authorizers.xml #nifi.login.identity.provider.configuration.file=/data05/nifi/conf/login-identity-providers.xml #nifi.templates.directory=/data05/nifi/conf/templates nifi.authorizer.configuration.file=/opt/nifi-1.9.2/conf/authorizers.xml nifi.login.identity.provider.configuration.file=/opt/nifi-1.9.2/conf/login-identity-providers.xml nifi.templates.directory=/opt/nifi-1.9.2/conf/templates nifi.ui.banner.text= nifi.ui.autorefresh.interval=30 sec nifi.nar.library.directory=./lib nifi.nar.library.autoload.directory=./extensions nifi.nar.working.directory=./work/nar/ nifi.documentation.working.directory=./work/docs/components #################### # State Management # #################### #nifi.state.management.configuration.file=./conf/state-management.xml nifi.state.management.configuration.file=/data05/nifi/conf/state-management.xml # The ID of the local state provider nifi.state.management.provider.local=local-provider # The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster. nifi.state.management.provider.cluster=zk-provider # Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server #nifi.state.management.embedded.zookeeper.start=false nifi.state.management.embedded.zookeeper.start=true # Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true #nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties nifi.state.management.embedded.zookeeper.properties=/data05/nifi/conf/zookeeper.properties # H2 Settings #nifi.database.directory=./database_repository nifi.database.directory=/data04/nifi/database_repo/prd nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE # FlowFile Repository nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog nifi.flowfile.repository.directory=/data02/nifi/flowfile_repo/prd nifi.flowfile.repository.partitions=256 nifi.flowfile.repository.checkpoint.interval=2 mins nifi.flowfile.repository.always.sync=false nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager nifi.queue.swap.threshold=20000 nifi.swap.in.period=5 sec nifi.swap.in.threads=1 nifi.swap.out.period=5 sec nifi.swap.out.threads=4 # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=1 MB nifi.content.claim.max.flow.files=100 nifi.content.repository.directory.default=/data03/nifi/cont_repo/prd nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=../nifi-content-viewer/ # Provenance Repository Properties nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository nifi.provenance.repository.debug.frequency=1_000_000 nifi.provenance.repository.encryption.key.provider.implementation= nifi.provenance.repository.encryption.key.provider.location= nifi.provenance.repository.encryption.key.id= nifi.provenance.repository.encryption.key= # Persistent Provenance Repository Properties nifi.provenance.repository.directory.default=/data01/nifi/provenance_repo/prd #nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.max.storage.time=3 days #nifi.provenance.repository.max.storage.size=1 GB nifi.provenance.repository.max.storage.size=10 GB #nifi.provenance.repository.rollover.time=30 secs nifi.provenance.repository.rollover.time=1 min #nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.rollover.size=512 MB nifi.provenance.repository.query.threads=2 #nifi.provenance.repository.index.threads=2 nifi.provenance.repository.index.threads=4 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are: # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type nifi.provenance.repository.indexed.attributes= # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository # but should provide better performance #nifi.provenance.repository.index.shard.size=500 MB nifi.provenance.repository.index.shard.size=2 GB # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved. nifi.provenance.repository.max.attribute.length=65536 nifi.provenance.repository.concurrent.merge.threads=2 nifi.provenance.repository.warm.cache.frequency=1 hour # Volatile Provenance Respository Properties nifi.provenance.repository.buffer.size=100000 # Component Status Repository nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository nifi.components.status.repository.buffer.size=1440 nifi.components.status.snapshot.frequency=1 min # Site to Site properties nifi.remote.input.host=ip-172-31-38-192.us-west-2.compute.internal #nifi.remote.input.secure=false nifi.remote.input.secure=true #nifi.remote.input.socket.port= nifi.remote.input.socket.port=9997 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec #nifi.remote.input.http.transaction.ttl=60 secs nifi.remote.contents.cache.expiration=30 secs # web properties # nifi.web.war.directory=./lib #nifi.web.http.host=ip-172-31-38-192.us-west-2.compute.internal #nifi.web.http.port=8080 nifi.web.http.network.interface.default= #nifi.web.https.host= nifi.web.https.host=ip-172-31-38-192.us-west-2.compute.internal #nifi.web.https.port= nifi.web.https.port=9696 nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= # security properties # nifi.sensitive.props.key= nifi.sensitive.props.key.protected= nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL nifi.sensitive.props.provider=BC nifi.sensitive.props.additional.keys= ##Through toolkit### nifi.security.keystore=/opt/nifi-1.9.2/conf/keystore.jks nifi.security.keystoreType=jks nifi.security.keystorePasswd=asHvHZPz6wkciFP89a30Aseb/HBF/oFjvpqtORhfdtA nifi.security.keyPasswd=asHvHZPz6wkciFP89a30Aseb/HBF/oFjvpqtORhfdtA nifi.security.truststore=/opt/nifi-1.9.2/conf/truststore.jks nifi.security.truststoreType=jks nifi.security.truststorePasswd=r+QbYY3qj/LfAxYXsXpjjse5b5ey8ZAKL72815byLkI nifi.security.needClientAuth=true nifi.security.user.authorizer=file-provider nifi.security.user.login.identity.provider=ldap-provider nifi.security.ocsp.responder.url= nifi.security.ocsp.responder.certificate= # OpenId Connect SSO Properties # nifi.security.user.oidc.discovery.url= nifi.security.user.oidc.connect.timeout=5 secs nifi.security.user.oidc.read.timeout=5 secs nifi.security.user.oidc.client.id= nifi.security.user.oidc.client.secret= nifi.security.user.oidc.preferred.jwsalgorithm= # Apache Knox SSO Properties # nifi.security.user.knox.url= nifi.security.user.knox.publicKey= nifi.security.user.knox.cookieName=hadoop-jwt nifi.security.user.knox.audiences= # nifi.security.group.mapping.pattern.anygroup=^(.*)$ # nifi.security.group.mapping.value.anygroup=$1 # nifi.security.group.mapping.transform.anygroup=LOWER # cluster common properties (all nodes must have same values) # #nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.heartbeat.interval=30 sec #nifi.cluster.protocol.is.secure=false nifi.cluster.protocol.is.secure=true # cluster node properties (only configure for cluster nodes) # #nifi.cluster.is.node=false nifi.cluster.is.node=true #nifi.cluster.node.address= nifi.cluster.node.address=ip-xxx.us-west-2.compute.internal nifi.cluster.node.protocol.port=9999 nifi.cluster.node.protocol.threads=30 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=50 sec nifi.cluster.node.read.timeout=50 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=3 # cluster load balancing properties # nifi.cluster.load.balance.host= nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=4 nifi.cluster.load.balance.max.thread.count=8 nifi.cluster.load.balance.comms.timeout=30 sec # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=ip-xxx.us-west-2.compute.internal:2181,ip-xx.us-west-2.compute.internal:2181,ip-1xx.us-west-2.compute.internal:2181 nifi.zookeeper.connect.timeout=30 secs nifi.zookeeper.session.timeout=30 secs nifi.zookeeper.root.node=/nifi # Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management # Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes # and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node # The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal # and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal # values configured on the zookeeper server). nifi.zookeeper.auth.type= nifi.zookeeper.kerberos.removeHostFromPrincipal= nifi.zookeeper.kerberos.removeRealmFromPrincipal= #nifi.zookeeper.auth.type=sasl #nifi.zookeeper.kerberos.removeHostFromPrincipal=true #nifi.zookeeper.kerberos.removeRealmFromPrincipal=true # kerberos # nifi.kerberos.krb5.file= # kerberos service principal # nifi.kerberos.service.principal= nifi.kerberos.service.keytab.location= # kerberos spnego principal # nifi.kerberos.spnego.principal= nifi.kerberos.spnego.keytab.location= nifi.kerberos.spnego.authentication.expiration=12 hours # external properties files for variable registry # supports a comma delimited list of file locations nifi.variable.registry.properties=   2) login-identity-provider.xml <loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">xxxxxx</property> <property name="Manager Password">xxx</property> <property name="TLS - Keystore"/> <property name="TLS - Keystore Password"/> <property name="TLS - Keystore Type"/> <property name="TLS - Truststore"/> <property name="TLS - Truststore Password"/> <property name="TLS - Truststore Type"/> <property name="TLS - Client Auth"/> <property name="TLS - Protocol"/> <property name="TLS - Shutdown Gracefully"/> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://xx.ad.xx.edu:xx</property> <property name="User Search Base">dc=xx,dc=xxx,dc=xx</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> </provider> </loginIdentityProviders>   3) authorizers.xml   <authorizers> <authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial Admin Identity">CN=Nifi_Admin,OU=NIFI</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=Nifi_Admin,OU=NIFI</property> <property name="Node Identity 2">CN=Nifi_Admin,OU=NIFI</property> <property name="Node Identity 3">CN=Nifi_Admin,OU=NIFI</property> </authorizer> </authorizers> ... View more

@Matt ... View more