@Faisal Durrani 1. As of HDF 2.x NiFI was redesigned to be a zero-master cluster service. Previously their was a single NiFI cluster Manager and a bunch of member nodes. HDF 2.x and on uses Zookeeper to elect a cluster coordinator and Primary Node form any one of he installed NiFi nodes. This allows provides a HA control plane in NiFI. If the current elected node for either of those roles goes down, ZK will elect an replacement form the remaining active nodes. 3 nodes is the recommended minimum NiFi cluster size, so that is where Ambari starts for you. You can install as many node as you like. 2. Zookeeper needs to have an odd number of hosts to maintain quorum. While Ambari installs ZK on the same servers/hosts as NiFi, you can re-assign the ZK instances to any hosts you like. You do not need a ZK on every host that has NIFi and they do not need to be co-located on the sam host. Even with a 9 node NIFi cluster, a 3-5 node ZK is fine. NiFI uses ZK for electing the Cluster Coordinator, electing the Primary Node, and to store cluster wide state (Many components in NIFi record state). 3. There is a single Cluster Coordinator and a single Primary node (They can change at any time). It is not uncommon to see a single NIFi node get elected to serve both roles. The Primary ode is responsible for running any processors configured for "primary node" only. Since every NiFi node/instance contains the exact same dataflows (stored on flow.xml.gz file), every node must be able o run and validate the configurations locally. So any directories or configuration files needed by NiFi components must exist on every node. In you case where you are consuming from a local file system, it may be bets to create a network disk that is identically mounted by every node in your NiFi cluster. You can then configure your NiFi ingest processor of choice (for example ListFile) to run on "primary node" only. That way should the primary node change, any node can access the same source of data and continue ingesting data where the previous elected primary node left off. Hope this addresses all your questions here. Thanks, Matt If you find this answer addressed your questions, please take a moment to click "accept" below the answer to mark this thread complete. ... View more

@Eric Lloyd I am assuming the file being tailed is mounted across all your NiFi nodes in the cluster? This would need to be the case so that no matter which node becomes the primary node, it could tail the exact same file. Assuming the above is true, I am also assuming processor has been configured with "State Location" configured for "Remote" When listFile executes it begins tailing the target File, at completion of each thread state is recorded as to where that tail let off so next thread can pickup where the previous ended. If you are only storing state "Local" when primary node switches, the new primary node will start tailing from beginning of file again. That being said, there is still a chance for some duplication even when state is set to Cluster. When primary node changes, original primary node is informed it is no longer the primary node and a new node is elected as the primary node. The original node will complete it currently executing task but will not schedule any new tasks to run. The New primary node will start the "primary node" only processors. If the new primary node executes before same processor on old primary node updates cluster state, it is possible new primary node will start tailing from last known recorded cluster state for that processor resulting in some duplication. NiFi favors duplication over data loss. We cannot assume that the original primary node just did not die. So we have to accept the risk that the original primary node processors may never update state. Hope this confirms how your processor is setup and why NiFi works the way it does in this scenario. Thanks, Matt ... View more

@Arun A K You will need to change the default authorizer in "Advanced nifi-authorizers-env" NiFi config section via Ambari. You will want to replace the default authorizer: <authorizer> <identifier>{{nifi_authorizer}}</identifier> {% if has_ranger_admin and enable_ranger_nifi %} <class>org.apache.nifi.ranger.authorization.RangerNiFiAuthorizer</class> <property name="Ranger Audit Config Path">{{nifi_config_dir}}/ranger-nifi-audit.xml</property> <property name="Ranger Security Config Path">{{nifi_config_dir}}/ranger-nifi-security.xml</property> <property name="Ranger Service Type">nifi</property> <property name="Ranger Application Id">nifi</property> <property name="Allow Anonymous">true</property> <property name="Ranger Admin Identity">{{ranger_admin_identity}}</property> {% if security_enabled %} <property name="Ranger Kerberos Enabled">true</property> {% else %} <property name="Ranger Kerberos Enabled">false</property> {% endif %} {% else %} <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">{{nifi_flow_config_dir}}/authorizations.xml</property> <property name="Users File">{{nifi_flow_config_dir}}/users.xml</property> <property name="Initial Admin Identity">{{nifi_initial_admin_id}}</property> <property name="Legacy Authorized Users File"></property> {% endif %} {{nifi_ssl_config_content}} </authorizer with the something similar to the following: <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN"></property> <property name="Manager Password"></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url"></property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="User Search Base"></property> <property name="User Object Class"></property> <property name="User Search Scope">ONE_LEVEL</property> <property name="User Search Filter"></property> <property name="User Identity Attribute"></property> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base"></property> <property name="Group Object Class"></property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute"></property> <property name="Group Member Attribute"></property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <authorizer> <identifier>ranger-authorizer</identifier> <class>org.apache.nifi.ranger.authorization.ManagedRangerAuthorizer</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Ranger Audit Config Path"><path to nifi conf dir>/ranger-nifi-audit.xml</property> <property name="Ranger Security Config Path"><path to nifi conf dir>/ranger-nifi-security.xml</property> <property name="Ranger Service Type">nifi</property> <property name="Ranger Application Id">nifi</property> <property name="Allow Anonymous">true</property> <property name="Ranger Admin Identity"></property> <property name="Ranger Kerberos Enabled">false</property> </authorizer> Details on the "ldap-user-group-provider can be found in the nifi admin guide: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.0/bk_administration/content/authorizers-setup.html Make sure the <userGroupProvider> comes before the <authorizer> in this configuration. Thanks, Matt ... View more