Configuring an external Zookeeper to work with NiFi stand alone mode

[#1]

Hello,

I have a question if you can help answer that would be great.

Can we run Nifi in standalone mode (not cluster) in an Open Shift Pod but manage state in zookeeper to ensure if the openshift pod reboots the states of the stateful processors are not lost? it appears to me that NiFi architecture does not fit containerization.

Thoughts?

[#2]

@Prachi Sharma

A standalone NiFi uses the configured nifi.state.management.provider.local for everything while in a clustered environment state can be stored in either the configured nifi.state.management.provider.local or nifi.state.management.provider.cluster.
-

I see no reason why you cannot just define a different state-provider for nifi.state.management.provider.local that uses ZK instead of local disk. This configuration change would be made in the state-management.xml file and nifi.properties file.
-

For a standalone NiFi, you could simply configure the nifi.state.management.provider.local= property in the nifi.properties file to point at the existing "zk-provider" in the state-management.xml file. You then want to edit the zk-provider in the state-management.xml file with the required ZK properties.

-

I recommend testing this out. In theory it should work perfectly, but i have not set this up myself.

-

Thanks,

Matt

-

When an "Answer" addresses/solves your question, please select "Accept" beneath that answer. This encourages user participation in this forum.

[#3]

@Prachi Sharma

-

*** Forum tip: Please try to avoid responding to an Answer by starting a new answer. Instead use the "add comment" tp respond to en existing answer. There is no guaranteed order to different answers which can make following a response thread difficult especially when multiple people are trying to assist you.

-

Please try creating a new "local-provider" rather then pointing at the "cluster-provider".

Comment out the following:

            
<!--
    <local-provider>
            <id>local-provider</id>
            <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
            <property name="Directory">/var/lib/nifi/state/local</property>
            <property name="Always Sync">false</property>
            <property name="Partitions">16</property>
            <property name="Checkpoint Interval">2 mins</property>
    </local-provider>
-->

Then add the following new "local-provider":

<local-provider>
           <id>zk-provider</id>
           <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
           <propertyname="Connect String">myhostname:myport</property>
           <propertyname="Root Node">/nifi</property>
           <propertyname="Session Timeout">10 seconds</property>
           <propertyname="Access Control">Open</property>
</local-provider>

-

Then change nifi.state.management.provider.local= in the nifi.properties file so it points back at "local-provider"

-

Thank you,

Matt

[#4]

@Prachi Sharma

-

Sorry to hear those possible solutions did not work.

That really just leave one option. Standing up a one node cluster.

-

I encourage you to open and Apache NiFi Jira requesting this capability be supported.

-

Once again, I am sorry that those suggestions did not work.

-
Thanks,

Matt

[#5]

@Matt Clarke

Hello, just wanted to update that this did not work. I changed the state-provider for nifi.state.management.provider.local to zk-provider in nifi.properties file. Then I updated the state-management.xml file with the ZK properties.

nifi.properties change is below:

# The ID of the local state provider
nifi.state.management.provider.local=zk-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider

<local-provider>
        <id>local-provider</id>
        <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
        <property name="Directory">./state/local</property>
        <property name="Always Sync">false</property>
        <property name="Partitions">16</property>
        <property name="Checkpoint Interval">2 mins</property>
 </local-provider>

<cluster-provider>
        <id>zk-provider</id>
        <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
        <property name="Connect String">myhostname:myport</property>
        <property name="Root Node">/nifi</property>
        <property name="Session Timeout">10 seconds</property>
        <property name="Access Control">Open</property>
</cluster-provider>

I have the correct hostname and port for zk-provider above in state-management.xml, I just have removed the entries in the post.

Question: Should be changing the root node?

Here is the error I am getting:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: Cannot create Local State Provider because the 'nifi.state.management.provider.local' property in the NiFi Properties file is set to 'zk-provider', but this id is assigned to a cluster-provider entry in the State Management Configuration File (/etc/nifi/state-management.xml), rather than a local-provider entry

Caused by: java.lang.IllegalStateException: Cannot create Local State Provider because the 'nifi.state.management.provider.local' property in the NiFi Properties file is set to 'zk-provider', but this id is assigned to a cluster-provider entry in the State Management Configuration File (/etc/nifi/state-management.xml), rather than a local-provider entry
	at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createStateProvider(StandardStateManagerProvider.java:157)
	at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createLocalStateProvider(StandardStateManagerProvider.java:87)
	at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.create(StandardStateManagerProvider.java:73)
	at org.apache.nifi.controller.FlowController.<init>(FlowController.java:507)
	at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:409)
	at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:77)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
	... 45 common frames omitted
2018-07-19 15:26:03,674 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-07-19 15:26:03,821 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@1ffe7c42{HTTP/1.1,[http/1.1]}{hdf-lab-0:8080}
2018-07-19 15:26:03,822 INFO [Thread-1] org.eclipse.jetty.server.session Stopped scavenging
Received trapped signal, beginning shutdown...

Is there other property settings I should use?

I would like to make NiFi work with containers.

Thanks.

[#6]

@Matt Clarke

This did not work either. First I couldn't use zk-provider in two places as id. It is already used in cluster-provider. I changed that as follows:

<local-provider>
           <id>local-provider</id>
           <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
           <property name="Connect String">myhostname:2181</property>
           <property name="Root Node">/nifi</property>
           <property name="Session Timeout">10 seconds</property>
           <property name="Access Control">Open</property>
</local-provider>

<cluster-provider>
        <id>zk-provider</id>
        <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
        <property name="Connect String">myhostname:2181</property>
        <property name="Root Node">/nifi</property>
        <property name="Session Timeout">10 seconds</property>
        <property name="Access Control">Open</property>
 </cluster-provider>

The error I get is as follow:

java.lang.RuntimeException: Cannot use Local State Provider (org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider) as it only supports scope(s) {CLUSTER} but instance is configured to use scope LOCAL
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
<br>

[#7]

@Matt Clarke

Another note, I tried Redis as well. That did not work as well. I pretty much get the same error. I will open an Apache NiFi ticket.

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cannot use Local State Provider (org.apache.nifi.redis.state.RedisStateProvider) as it only supports scope(s) {CLUSTER} but instance is configured to use scope LOCAL

Thanks for helping troubleshoot this.

I will explore my next steps now perhaps a VM of NiFi cluster.