@Hans Feldmann The individual processors allow for concurrent task changes. By default they all have one concurrent task. For each additional concurrent task, you are giving that processor the opportunity to request an additional thread from the NiFi controller to do work in parallel. (Think of it as two copies of the same processor doing working different files or batches of files). If there isn't sufficient files in the incoming queue then any additional concurrent tasks are not utilized. The flip side is if you allocate two many concurrent tasks to a single processor, that processor may itself end up using two many threads from the NiFi controller's resource pull resulting in a thread starvation to other processors. So star with the default and setup by one increment at a time in place of backlog in your flow. The NiFi controller also has a setting that limits the maximum number of threads it can use from the underlying hardware. This is the other thing Andrew was mentioning. A restart of NiFi is NOT needed when you make changes to these values. The defaults are low (10 timer driven and 5 event driven). I would set the timer driven to no more then double the number of cores your hardware has. Thanks, Matt ... View more

@Obaid Salikeen Not sure what "issues" you had when you tried to add a new node to your existing cluster. The components (processors, connections, etc...) of an existing cluster can be running when you add new additional nodes to it. The new nodes will inherit the flow and templates from the NCM as well as the current running state of those components when it joins. But, in order for a node to successfully join a cluster the following must be true: 1. The new node either has no flow.xml.gz file and templates directory or the flow.xml.gz file and templates do not match what is currently on the NCM. (Remove flow.xml.gz file and templates dir from new node and restart node) The nifi-app.log will indicate in if a difference was found. 2. The nifi.sensitive.props.key= in the nifi.properties file must have the same value as on the NCM. 3. The NCM must be able to resolve the URL to the new node. If the nifi.web.http(s).host= was left blank on your new node, Java on that node may be reporting the hostname as localhost. Make sure valid resolvable hostnames are supplied for nifi.web.http.host=, nifi.cluster.node.address=, and nifi.cluster.node.unicast.manager.address=. 4. Both NCM and Node security protocol must match. nifi.cluster.protocol.is.secure= in nifi.properties file. 5. Firewalls must be open between NCM and Node on both HTTP(s) port and node and NCM ports. 6. New node must have all the same available java classes. If custom processors exist in your flow make sure the new node also has those custom nar/jar files included in its lib dir. Thanks, Matt ... View more

@Saikrishna Tarapareddy Where all 74 files in the input queue before the MergeContent was run? The mergeContent processor just like the other processors works on a run schedule. My guess is that last file was not in the queue at the moment the MergeContent processor ran, so you only saw 13 get bundled instead of 14. With a min of 4 entries, it will read what is on the queue and bin it. You likely ended up with 3 bins with 20 and 1 bin with 13 because at the moment it looked at the queue 73 or 13 FlowFiles is all it saw. You can confirm this by stopping the MergeContent and allowing all 74 files to queue before staring it. The behavior should then be as you suspect. Sounds like it is not important to have exactly 20 per merged file. Perhaps you can set a max bin age so that files don't get stuck. Something else you can do is adjust the run schedule so the mergeContent does not run as often. The default is "o sec" which means run as fast as possible. Try changing that to somewhere between 1 and 10 sec to give the files a chance to queue. If you are picking up all the 74 files at the same time, we are likely talking milliseconds here that is causing this last file to get missed. Thanks, Matt ... View more

@Yogesh Sharma Is your NiFi a cluster or Standalone instance of NiFi? If it is a cluster, it could explain why you are seeing duplicates since the same GetTwitter processor would be running on every Node. Matt ... View more

@Ankit Jain When A NiFi instance is designated as a node its starts sending out heartbeat messages after it is started. Those heartbeat messages contain important connection information for the Node. Part of that messages is the hostname for each connecting node. If left blank Java will try to determine the hostname and in many cases the hostname ends up being "localhost". This may explain why the same configs worked when all instances where on the same machine. Make sure that all of the following properties have been set on everyone of your Nodes: # Site to Site properties nifi.remote.input.socket.host= <-- Set to the FQDN for the Node musty be resolvable by all other instances. nifi.remote.input.socket.port= <-- Set to unused port on Node. # web properties # nifi.web.http.host= <-- set to resolvable FQDN for Node nifi.web.http.port= <-- Set to unused port on Node # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address= <-- set to resolvable FQDN for Node nifi.cluster.node.protocol.port= <-- Set to unused port on Node nifi.cluster.node.protocol.threads=2 # if multicast is not used, nifi.cluster.node.unicast.xxx must have same values as nifi.cluster.manager.xxx # nifi.cluster.node.unicast.manager.address= <-- Set to the resolvable FQDN of your NCM nifi.cluster.node.unicast.manager.protocol.port= <-- must be set to Manager protocol port assigned on your NCM. Your NCM will need to be configured the same way as above for the Site-to-Site properties and Web properties, but instead of the "Cluster Node properties", you will need to fill out the "cluster manager properties": # cluster manager properties (only configure for cluster manager) # nifi.cluster.is.manager=true nifi.cluster.manager.address= <-- set to resolvable FQDN for NCMnifi.cluster.manager.protocol.port= <-- Set to unused port on NCM. The most likely cause of your issue is not having the host/address fields populated or trying to use a port that is already in use on the server. If setting the above does not resolve your issue, try setting DEBUG for the cluster logging in the logback.xml on one of your nodes and the NCM to get more details: <logger name="org.apache.nifi.cluster" level="DEBUG"/> ... View more