Member since
06-03-2021
3
Posts
0
Kudos Received
0
Solutions
06-04-2021
12:50 PM
Hi @MattWho , Thanks for your clear answer. I'd like understand what happens if we use a processor that supports only the "cluster" state on one (or many) standalone Nifi instance(s). Will the state be persisted locally or will it be ignored ? Thanks.
... View more
06-03-2021
03:17 AM
Hi, According to the official Nifi documentation, the state allows Nifi processors to "resume from the place where it left off after NiFi is restarted. Additionally, it allows for a Processor to store some piece of information so that the Processor can access that information from all of the different nodes in the cluster". If my understanding is good, when we configure a zookeeper Provider, the state will not be persisted locally, instead, the data will be sent to zookeeper. I've explored the zookeeper znodes and could not find any data related to the state, all I can find are the informations about the Coordinator and Primary nodes. However, the local state directory is still filled. The configuration is very simple, I've 3 external ZK nodes and 3 Nifi instances. Here is an exerpt of the nifi.properties file: nifi.cluster.is.node=true
nifi.zookeeper.connect.string=zk-node1:2181,zk-node2:2181,zk-node3:2181
nifi.state.management.embedded.zookeeper.start=false
nifi.state.management.provider.cluster=zk-provider And here is an exerpt of the state-management.xml file: <cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Connect String">zk-node1:2181,zk-node2:2181,zk-node3:2181</property>
<property name="Root Node">/nifi</property>
<property name="Session Timeout">10 seconds</property>
<property name="Access Control">Open</property>
</cluster-provider> When I try to ls the Zookeeper, I can see only 2 znodes: "components" but this znode is empty and the "leaders" zonde which contain some data about the Nifi Coordinator and Primary Nodes. Also, when I explore the transactions logs, even after using some load balanced connections, I cannot find anything related to the Nifi State. Could somebody explain what data goes the Zookeeper and why the local state directory is still filled even if we configure the zk provider ? Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Zookeeper