@Anwaar Siddiqui With Nifi running as http (non-secure) there is no way to differentiate between users who access the UI. To NiFi everyone is the same anonymous user. Two or more people can still work on their own dataflows within a non-secured NiFi, but there is no way to prevent each user from modifying another users dataflow. Once NiFi is running as https (secured), some mechanism must be but in place to authenticate the users who will be accessing the canvas. Currently supported user authentication methods include TLS user certificates (default), LDAP, or kerberos. Through user authentication NiFi can now distinguish between users. After authentication comes authorization. Authorization is handled by NiFi (default) or Apache Ranger. This authorization layer is used to grant specific access policies to specific authenticated users. Things like controlled access to specific components (processors, process groups, controller services, etc.) are handled through this authorization. While every user still access the same canvas, this allows you to control what components can be seen and modified down to a specific user if desired. Thanks, Matt ... View more

@bala krishnan When the MergeContent processor runs (based on run schedule configuration), it looks at the FlowFiles on the incoming queue and groups them in bins (think of these as logical containers) based on processor properties configurations. You have two Merge Strategies to choose from: 1. Defragmentation: This strategy requires that FlowFiles have the following attributes set on them. ------------------------------ *** Each "bundle" will go in to its own bin. ----------------------------- 2. Bin-Packing Algorithm (default): FlowFiles are simply placed in bins based on the criteria discussed below. Here are the properties that affect how the bins FlowFile are placed in are handled: -------------- *** FlowFile content is never truncated. For example: If a FlowFile where its content is larger then the configured "maximum Group Size" exists on incoming queue will simply become a merged file of one and passed directly to the merged relationship. -------------- *** No maximum simply means that there is no ceiling on the "maximum number of entries" or "maximum group size". Does this mean a bin will never get merged? NO. The currently queued FlowFiles on an incoming connection are placed in bin(s). If at the end of being placed in those bins, any of the bins has reached or exceeded the "minimum number of entries" or "minimum group size", that bin(s) is merged. -------------- *** Don't forget the "Max bin age" acts as your trump card. A bin will be merged if it has been around this long without being merged regardless of the other settings. -------------- Thanks, Matt ... View more

@Joshua Adeleke First lets make sure we are on the same page terminology wise.... NiFi FlowFiles --> FlowFiles are made up of two parts, FlowFile Content and FlowFile Attributes. FlowFiles Content is written to NiFI's content Repository while FlowFile Attributes mostly live in JVM heap memory and the NiFi FlowFile repository. It is the FlowFile attributes that move from processor to processor in your dataflow. Apache NiFi does not have a version 1.0.0.2.0.0.0-579. That is an HDF version of Apache NiFi 1.0.0. If you are migrating to HDF 2.0, I suggest instead migrating to HDF 2.0.1. It has many bug fixes you will want... https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/index.html When you say you want to move the "dataflow process: to another server, are you talking about your entire canvas and all configured processors only? Or are you also talking about moving any active FlowFiles within the old flow to the new? My suggestion would be to stand-up your new NiFi instance/cluster. You can copy your flow.xml.gz file from your old NiFi to the new NiFi. ***NOTE: you need to make sure that the same sensitive props key is used (config setting for this is found in nifi.properties file) or your new NiFi will not start because it will not be able to decrypt any of your sensitive properties in that file. If your old NiFi was secured, you can use its authorized-users.xml file to establish the initial admin authorities in the new NiFi (configure this in authorizers.xml file). Once started you will need to access this new NiFi's UI and address any "invalid" processor/controller services, and add any controller services you may have had running on the NCM only in the old version. (No NCM in HDF 2.x versions). Some of these may be invalid because of changes to the processor/controller service properties. Once all has been addressed you are ready to move to the next step. Shutdown all ingest processor on your old NiFi and allow it to finish processing out any data it is already working on. At the same time you can start your new NiFi NiFi so it starts ingesting any new data and begins processing it. Thanks, Matt ... View more

@Saikrishna Tarapareddy I agree that you may still need to split your very large incoming FlowFile into smaller FlowFiles to better manage heap memory usage, but you should be able to use the RouteText and ExtractText as follows to accomplish what you want: RouteText configured as follows: All Grouped lines will be routed to relationship "TagName" as a new FlowFile. They feed into an ExtractText configured as follows: This will extract the TagName as an attribute of on the FlowFile which you can then use as the correlationAttribute name in the MergeContent processor that follows. Thanks, Matt ... View more