@basant sinha I am not sure I am completely following your use case. With a NiFi cluster configuration it does not matter which nodes UI you access, the canvas you are looking at is what is running on all nodes in your cluster. The Node(s) that are the currently elected cluster coordinator and/or primary node may change at any time. A Remote Process Group (RPG) is used to send or retrieve FlowFiles from another NiFi instance/cluster. It is not used to list files on a system. So I am not following you there. When it comes to configuring a RPG, not all fields are required. 39917-screen-shot-2017-10-23-at-102418-am.png - You must provide the URL of the target NiFi instance/cluster (with a cluster, this URL can be anyone of the nodes). - You must choose either "RAW" (default) or "HTTP" as you desired Transport Protocol. No matter which is chosen, the RPG will connect to the target URL over HTTP to retrieve Site-To-Site (S2S) details about the target instance/cluster. (number of nodes if cluster, available remote input/output ports, etc...). When it comes to the actual data transmission: ----- If configured for "RAW" (which uses a dedicated S2S port configured in the nifi.properties property nifi.remote.input.socket.port on target NiFi), data will be sent over that port. ----- If configured for "HTTP", data will be transmitted over same port used in the URL for the target NiFi's UI. (This requires that nifi.remote.input.http.enabled property in nifi.properties file is set to "true") The other properties are optional: - Configure the Proxy properties if an external proxy server is sitting between your NiFi and the target NiFi preventing any direct connection between your NiFi instance over the above used configured ports (defaults: unset). Since NiFi RPG will be sending data directly to each target node (if target is cluster), none of the NiFi nodes themselves are acting as a proxy in this process. - Configure the Local Network Interface, if your NiFi nodes have multiple network ports and you want to force your RPG to only use a specific interface (default: unset). Additional documentation resources from Apache: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site_to_site_properties If you find this answer addresses your question, please take a moment to click "Accept" below the answer. Thanks, Matt ... View more

@dhieru singh I am not clear on what you mean by "stopping a dataflow resulted in data loss"? NiFi does not delete any data when a dataflow is stopped. Data will remain queued between the stopped components until the dataflow is restarted or a user manual operation is performed to purge the data from those queues. There is no notion of a "lock" in NIFi that can be set on a component or set of components in NiFi. In addition, having a a double confirm every time a user wants to stop a component to make an edit may be more annoying then beneficial. That being said, it might be an interesting idea to add the ability to "lock" the current running state of "process group". Basically putting all components in a process group in to read only mode until the lock is removed. Might be worth you creating an NiFi Apache Jira for such a thing. If you Nifi is secured, you can prevent such issues by taking away users "modify" access to the components. Without Modify access policies, users can only view the components. They will not be able to change active state (start, stop, enable, or disable) or the configuration. But this would also require that you re-add "modify" anytime a change is desired. Thank you, Matt ... View more

@Ben Morris NiFi has not explicitly defined max for the number of nodes that can be added to a single NiFi cluster. Just keep in mind that the more nodes you add, the more request replication that must occur between nodes. For example, If a user is connected to node 1 of 100 nodes and makes a change, that change must be replicated to all 99 other nodes. NiFi is configured with a number of node protocol threads (default 10). So NIFi is only capable of replicating that change to 10 nodes at a time. This value should be increased to accommodate larger clusters. Failing to adjust this value my result in nodes disconnecting because they did not receive the change request fast enough. In addition, you may need to be more tolerant on your connection and heartbeat timeouts. As far as max data per second, that is a hard number to lay out. It is highly dependent on a number of factors. Mostly affected by your particular dataflow implementation. Since NiFi is just a blank canvas in which you build your dataflow, in the end your dataflow design defines your performance/throughput in most cases. This comes down to which processors you use and how they are configured. Assuming you have a well designed and optimized dataflow design, you can expect upwards of the following: 39876-screen-shot-2017-10-19-at-84154-am.png *** These numbers will still be affected by use of some processors. CompressContent for example: this processor can be CPU intensive over longer periods of time when compressing large files, so I can become a bottleneck. If you found that this answer addressed you question, please take a moment to click "accept". Thank you, Matt ... View more

@Kiem Nguyen In a NiFi cluster, NiFi wants to make sure consistency across all nodes. You can't have each node in a NiFi cluster running a different version/state of the flow.xml.gz file. In a cluster, NiFi will replicate a request (such as stop x processor(s)) to all nodes. Since a node is not connected, that replication cannot occur. So to protect the integrity of the cluster, the NiFi canvas is essentially read-only while a node is disconnected. Your two options are: 1. Reconnect the disconnected node and then stop your dataflow(s). 2. Drop the disconnected node form your cluster via the "cluster" UI found in the hamburger menu in the upper right corner of the UI. This will make your cluster a 2 of 2 cluster and will return UI to full functionality. You will need to then restart that dropped node in order to get it to try to join the cluster again once fixed. Thanks, Matt ... View more

@Hadoop User Your Java regular expression needs to escape the "[" and "]" since they have reserved meaning in Java. Try using the following java regular expression instead: (\[hdfs.*log"\]) Thanks, Matt ... View more