issues while setting up Nifi Secure cluster version 1.0.0

[#1]

Hi, @Bryan Bende i am following below post to set nifi cluster -

http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy

Getting this error - while starting one of the nodes in the cluster . looks like its a recursive loop. Can you please help troubleshoot this ?

Thanks!

Juthika

o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/app_2/runtime/nifi/./conf/nifi.properties' 2016-10-19 08:50:18,653 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/app_2/runtime/nifi/./conf/nifi.properties' 2016-10-19 08:50:18,654 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 116 properties from /app_2/runtime/nifi/./conf/nifi.properties 2016-10-19 08:50:48,301 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE. Building now... 2016-10-19 08:50:48,595 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded 102 properties from system properties and environment variables 2016-10-19 08:50:48,597 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded 11 properties from './conf/coda.properties' 2016-10-19 08:50:48,598 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded a total of 113 properties. Including precedence overrides effective accessible registry key size is 113 2016-10-19 08:50:48,697 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 274719330795 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2016-10-19 08:50:48,700 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2016-10-19 08:50:49,082 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@3e7940b3 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2016-10-19 08:50:49,082 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 14 milliseconds 2016-10-19 08:50:49,103 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@3e7940b3 checkpointed with 0 Records and 0 Swap Files in 20 milliseconds (Stop-the-world time = 4 milliseconds, Clear Edit Logs time = 3 millis), max Transaction ID -1 2016-10-19 08:50:49,180 INFO [main] o.a.n.c.s.server.ZooKeeperStateServer Starting Embedded ZooKeeper Peer 2016-10-19 08:50:49,250 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2016-10-19 08:50:49,324 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2016-10-19 08:50:56,269 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected 2016-10-19 08:50:56,270 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2016-10-19 08:50:56,378 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2016-10-19 08:50:56,379 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2016-10-19 08:50:56,380 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2016-10-19 08:50:56,384 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2016-10-19 08:50:56,384 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2016-10-19 08:50:56,384 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2016-10-19 08:50:56,434 WARN [main] o.eclipse.jetty.util.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlets.GzipFilter 2016-10-19 08:50:56,435 WARN [main] org.eclipse.jetty.servlets.GzipFilter GzipFilter is deprecated. Use GzipHandler 2016-10-19 08:50:56,438 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5529fd4e{/nifi-api,file:///app_2/runtime/nifi/work/jetty/nifi-web-api-1.1.0-SN...} 2016-10-19 08:50:57,106 INFO [main] /nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2016-10-19 08:50:57,132 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@7dbc77ca{/nifi-content-viewer,file:///app_2/runtime/nifi/work/jetty/nifi-web-c...} 2016-10-19 08:50:57,134 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@184e5c44{/nifi-docs,null,AVAILABLE} 2016-10-19 08:50:57,199 INFO [main] /nifi-docs No Spring WebApplicationInitializer types detected on classpath 2016-10-19 08:50:57,201 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6527aa0{/nifi-docs,file:///app_2/runtime/nifi/work/jetty/nifi-web-docs-1.1.0-S...} 2016-10-19 08:50:57,238 INFO [main] / No Spring WebApplicationInitializer types detected on classpath 2016-10-19 08:50:57,261 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6cbb79c3{/,file:///app_2/runtime/nifi/work/jetty/nifi-web-error-1.1.0-SNAPSHOT...} 2016-10-19 08:50:57,269 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@13157620(coda-nifi-ssl-cert,h=[apsrt3387.ccc.com, apsrt3389.ccc.com, apsrt3388.ccc.com, apsrt3391.ccc.com, apsrt3390.ccc.com, apsrt3402.ccc.com, apsrt3395.ccc.com, apsrt3394.ccc.com, apsrt3393.ccc.com, apsrt3396.ccc.com, apsrt3398.ccc.com, apsrt3397.ccc.com, apsrt3399.ccc.com, apsrt3409.ccc.com, apsrt3408.ccc.com, apsrt3403.ccc.com, apsrt3400.ccc.com, apsrt3401.ccc.com, apsrt3410.ccc.com, ccc.com],w=[]) for SslContextFactory@62a78446(file:///app_2/runtime/nifi/conf/coda-nifi-ssl-cert.pfx,file:///app_2/runt...) 2016-10-19 08:50:57,289 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@361b2995{SSL,[ssl, http/1.1]}{apsrt3390.ccc.com:8443} 2016-10-19 08:50:57,289 INFO [main] org.eclipse.jetty.server.Server Started @87585ms 2016-10-19 08:50:58,372 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow... 2016-10-19 08:50:58,380 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 9443 2016-10-19 08:50:58,442 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: apsrt3390.ccc.com:8443 2016-10-19 08:51:05,041 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2016-10-19 08:51:05,041 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2016-10-19 08:51:13,776 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2016-10-19 08:51:13,776 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2016-10-19 08:51:14,013 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2016-10-19 08:51:14,015 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@15cca12c Connection State changed to SUSPENDED 2016-10-19 08:51:14,019 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) [curator-framework-2.11.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) [curator-framework-2.11.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_65] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] 2016-10-19 08:51:14,020 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838) [curator-framework-2.11.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300

[#2 · Accepted solution]

That means the user you are logging in as does not have permission to access the UI. You can check nifi-user.log to see the user identity that is coming from your request (it should be the DN of your cert) and compare that to what is in users.xml and authorizations.xml.

If this is your "initial admin" identity then this should have been entered in authorizers.xml as the initial admin, and that would have granted it all the correct permissions. If you had already tried to setup an initial admin before then you need to delete users.xml and authorizations.xml before trying to change the "initial admin", otherwise it won't take effect.

[#3]

I updated according to the log , everything works perfectly now - Thanks very much for your help

Juthika

[#4]

@Juthika Shenoy

This error indicates and authorization issue. This is separate from authentication. I would start by looking at your nifi-user.log and see what DN is successfully authenticating by being denied authorization. Then verify that DN is included along with your node identity(s) in the users.xml file. If it is not, then that is your problem.

I noticed from your post above you never provided an "Initial Admin Identity" in your authorizers.xml file. This is a must in order to get an initial admin added to the system so that that initial admin can then add additional users via the UI.

You can take your user DN from the nifi-user.log and add it to your authorizers.xml file:

<property name="Initial Admin Identity">Add user DN Here</property>

Also make sure you still have your Node Identity(s) set in the authorizers.xml file as well.

<property name="Node Identity 1">DN From Node 1 Cert Here</property>
<property name="Node Identity 2">DN From Node 2 Cert Here</property>
etc.....

If every Node is using the same cert. That cert must have a Subject Alternative Name (SAN) entry for each nodes FQDN. From a security standpoint, it is not recommended using one cert for multiple servers.

Finally, you will need to stop your NiFi nodes, delete your existing users.xml and authorizations.xml files form each of them, and then restart. NiFi will only create those tow files once. Once they have been created, changes to the authorizers.xml file will not trigger updates to them.

Thanks,

Matt