Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi Site to Site Configuration

Nifi Site to Site Configuration

New Contributor

Hi All,

Can some one please help me with steps on how to configure the site to site protocol in Nifi.

We have 3 node hadoop cluster and Nifi installation on one node.Currently , Nifi is running on standalone mode and I want it to run on all node for load balancing.

Thanks,

Sumit

3 REPLIES 3

Re: Nifi Site to Site Configuration

New Contributor

Hi @Sumit Deshmukh. There are some fantastic articles on cluster configuration by @Pierre Villard for both a non-secure (https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/) and secure environment (https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/). Likewise, @Bryan Bende has another excellent article on secure S2S configuration which will guide you through the process (https://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site). If load balancing is what you require I'd suggest that a cluster configuration is probably a better fit for your need. Finally, you may want to consider a load balancer as recommended by @kkawamura (https://community.hortonworks.com/articles/68375/nifi-cluster-and-load-balancer.html)

Re: Nifi Site to Site Configuration

New Contributor

Thanks D H for sharing the links.

I have started created the unsecure Nifi cluster with 3 nodes by referring the above link (https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/), made the necessary changes in nifi.properties.

I am getting below error.

Can you please me.

org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:93) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.Server.start(Server.java:411) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.Server.doStart(Server.java:378) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:651) ~[nifi-jetty-1.0.0.jar:1.0.0] at org.apache.nifi.NiFi.<init>(NiFi.java:152) [nifi-runtime-1.0.0.jar:1.0.0] at org.apache.nifi.NiFi.main(NiFi.java:243) [nifi-runtime-1.0.0.jar:1.0.0] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: Unable to initailize Flow because NiFi was configured to start an Embedded Zookeeper server but failed to do so at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:52) ~[na:na] ... 23 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: Unable to initailize Flow because NiFi was configured to start an Embedded Zookeeper server but failed to do so at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] ... 29 common frames omitted Caused by: java.lang.IllegalStateException: Unable to initailize Flow because NiFi was configured to start an Embedded Zookeeper server but failed to do so at org.apache.nifi.controller.FlowController.<init>(FlowController.java:566) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:405) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:61) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE] ... 36 common frames omitted Caused by: java.io.IOException: Failed to start embedded ZooKeeper Peer at org.apache.nifi.controller.state.server.ZooKeeperStateServer.startDistributed(ZooKeeperStateServer.java:126) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.state.server.ZooKeeperStateServer.start(ZooKeeperStateServer.java:62) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.FlowController.<init>(FlowController.java:564) ~[nifi-framework-core-1.0.0.jar:1.0.0] ... 39 common frames omitted Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_91] at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_91] at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_91] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[na:1.8.0_91] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.8.0_91] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67) ~[na:1.8.0_91] at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.nifi.controller.state.server.ZooKeeperStateServer.startDistributed(ZooKeeperStateServer.java:104) ~[nifi-framework-core-1.0.0.jar:1.0.0] ... 41 common frames omitted 2017-09-06 12:11:19,750 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2017-09-06 12:11:19,757 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@23243bc0{HTTP/1.1,[http/1.1]}{10.62.137.222:9443} 2017-09-06 12:11:19,763 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@55e02661{/,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.0.0.war} 2017-09-06 12:11:19,765 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@40c92dfa{/nifi-docs,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.0.0.war} 2017-09-06 12:11:19,765 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.s.h.ContextHandler@31dbca5d{/nifi-docs,null,UNAVAILABLE} 2017-09-06 12:11:19,767 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@4e77f29e{/nifi-content-viewer,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.0.0.war} 2017-09-06 12:11:19,768 INFO [Thread-1] o.a.n.w.c.ApplicationStartupContextListener Initiating shutdown of flow service... 2017-09-06 12:11:19,806 INFO [Thread-1] o.a.n.p.PersistentProvenanceRepository Recovered 0 records 2017-09-06 12:11:19,817 INFO [Thread-1] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0 2017-09-06 12:11:19,826 INFO [Thread-1] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 27910929612 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2017-09-06 12:11:19,826 INFO [Thread-1] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2017-09-06 12:11:19,916 INFO [Thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@7ca74a2e finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2017-09-06 12:11:19,916 INFO [Thread-1] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 4 milliseconds 2017-09-06 12:11:19,922 INFO [Thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@7ca74a2e checkpointed with 0 Records and 0 Swap Files in 5 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1 2017-09-06 12:11:19,926 INFO [Thread-1] o.a.n.c.s.server.ZooKeeperStateServer Starting Embedded ZooKeeper Peer 2017-09-06 12:11:19,927 WARN [Thread-1] o.a.n.w.c.ApplicationStartupContextListener Problem occurred ensuring flow controller or repository was properly terminated due to org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: Unable to initailize Flow because NiFi was configured to start an Embedded Zookeeper server but failed to do so 2017-09-06 12:11:19,927 INFO [Thread-1] /nifi-api Closing Spring root WebApplicationContext 2017-09-06 12:11:19,975 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@23a78d2e{/nifi-api,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.0.0.war} 2017-09-06 12:11:20,027 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@6393bf8b{/nifi,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.0.0.war} 2017-09-06 12:11:20,031 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@1084ac45{/nifi-image-viewer-1.0.0,null,UNAVAILABLE}{./work/nar/extensions/nifi-media-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.0.0.war} 2017-09-06 12:11:20,033 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@2091833{/nifi-standard-content-viewer-1.0.0,null,UNAVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-standard-content-viewer-1.0.0.war} 2017-09-06 12:11:20,036 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@6cbe68e9{/nifi-update-attribute-ui-1.0.0,null,UNAVAILABLE}{./work/nar/extensions/nifi-update-attribute-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-update-attribute-ui-1.0.0.war} 2017-09-06 12:11:20,041 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@399ca607{/nifi-jolt-transform-json-ui-1.0.0,null,UNAVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.0.0.war} 2017-09-06 12:11:20,049 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).

Highlighted

Re: Nifi Site to Site Configuration

New Contributor

@Sumit Deshmukh can you please attach your nifi.properties and zookeeper.properties files? (masking any sensitive values)