@SandyClouds  @steven-matison response if perfect. Just to add to your second question "2. NiFi jobs also start automatically without me running from UI ?"   NiFi is really designed as an always on data in motion service.  NiFi preserves the state (stopped, started, enabled, disabled) of all its components in the flow.json.gz file.  On startup of the service, NiFi will load the flow.json.gz, load FlowFiles form the NiFi Repository back in to connection within your dataflow, and return all components to last known state.  There is a property in the nifi.properties file "nifi.flowcontroller.autoResumeState=true" that controls this behavior.  If you change it from default "true" to "false", NiFi will reset all components to stopped (processors) or disabled (controller services and reporting tasks) on startup. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt   ... View more

@SandyClouds  Some clarity and additions to @cotopaul Pros and Cons: Single Node: PROs: - easy to manage.  <--  Setup and managing configuration is easier since you only need to do that on one node.  But in a cluster, all nodes configuration files will be almost the same (some variations in hostname properties and certificates if you secure your cluster). - easy to configure.  <-- There are more configurations needed in a cluster setup, but once setup, nothing changes from the user experience when it comes to interacting with the UI. - no https required.   <-- Not sure how this is a PRO.  I would not recommend using an un-secure NiFi as doing so allow anyone access to your dataflows and the data being processed.  You can also have an un-secure NiFi cluster while i do not recommend that either.   CONs: - in case of issues with the node, you NiFi instance is down.  <-- Very true, single point of failure. - it uses plenty of resources, when it needs to process data, as everything is done on a single node. Cluster: PROs: - redundancy and failover --> when a node goes down, the others will take over and process everything, meaning that you will not get affected.  <-- Not complete accurate.  Each node in a NiFi cluster is only aware of the data (FlowFiles) queued on that specific node.  So each node works on the FlowFile present on that one node, so it is the responsibility of the dataflow designer/builder to make sure they built their dataflows in such away to ensure distribution of FlowFiles across all nodes.  When a node goes down, any data FlowFiles currently queued on that down node are not going to be processed by the other nodes.  However, other nodes will continue processing their data and all new data coming in to your dataflow cluster - the used resources will be split among all the nodes, meaning that you can cover more use cases as on a single node.  <-- Different nodes do not share or pool resources from all nodes in the cluster.  If your dataflow(s) are built correctly the volume of data (FlowFiles) being processed will be distributed across all your nodes along each node to process a smaller subset of the overall FlowFile volume.  This means more resources available across yoru cluster to handle more volume.   NEW -- A NiFi cluster can be accessed via any one of the member nodes.  No matter which node's UI you access, you will be presented with stats for all nodes.  There is a cluster UI accessible from the global menu that allows you to see a breakdown of each node. Any changes you make from the UI of any one of the member nodes will be replicated to all nodes.   NEW -- Since all nodes run their own copy of the flow, a catastrophic node failure does not mean loss of all your work since the same flow.json.gz (contains everything related to your dataflows) can be retrieved from any of the other nodes in your cluster. CONs: - complex setup as it requires a Zookeeper + plenty of other config files.  <-- NiFi cluster requires a multi node zookeeper setup.  Zookeeper quorum is required for cluster stability and also stores cluster  wide state needed for your dataflow. Zookeeper is responsible for electing a node in your cluster with the Cluster Coordinator role and Primary node role.  IF a node goes down that has been assigned one of these roles, Zookeeper will elected one of the still up nodes to the role - complex to manage --> analysis will be done on X nodes instead of a single node.  <-- not clear.  Yes you have multiple nodes and all those nodes are producing their own set of NiFi-logs.  However, if a component within your dataflow is producing bulletins (exceptions) it will report all nodes or the specific node(s) on which bulletin was produced.    Cloudera offers centralized management of your NiFi cluster  deployment via Cloudera Manager software.  Makes deploying and managing NiFi cluster to multiple nodes easy, sets up and configures Zookeeper for you, and makes securing your NiFi easy as well by generating the needed certificates/keystores for you.   Hope this helps, Matt ... View more

@VLban  MergeContent and MergeRecords handling merging of FlowFiles's content differently.  Since your FlowFiles already contain Json formatted record(s), using MergeContent is not going to be the correct processor to use.  MergeContent does not care about the data/content format (except for Avro) of the inbound FlowFiles.  With Binary Concatenation, On flowFile's content bytes are simply write starting at the end of the last FlowFile's content. So in the case of JSON, the resulting merged FlowFile's content is not going to be valid json anymore.   Both processors will bin FlowFiles each time the Processor executes based on its run schedule.  At the end of each bin cycle the bins are evaluated to see if both configured mins are satisfied.  If so, the bin will be merged.  Setting a max does not mean that the bin will wait to get merged until the max has been met.  So you would be better to set your min to 500 MB if you always want files of at least 500 MB and set you max to a value a bit larger then that.  Doing so may result in bins that say have 480 MB binned and next FlowFile can't be added because it would then exceed configured max (FlowFile placed in new bin).  So the Max Bin Age property when set will force a bin to merge once the bin has existed for the configured max bin age (this avoid FlowFile getting stuck in these merge based processors).   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more