@sho aa Based on your first screenshot provided, it appears the putHbaseJson processor is processing FlowFiles. The stats indicate that ~783,000 FlowFiles had been pulled from the inbound connection queue in the last 5 minutes. The PutHbaseJson processor also shows that it has an active thread (1) at the time of the screenshot. It appears you are auto-terminating both the success and failure outbound relationships on your PutHbaseJson processor. If FlowFiles were auto-terminating on the Failire relationship, you should be seeing a Red square (bulletin) which your screenshot does not show. The bulletin reports an error log was produced. So I can only assume if you are not seeing bulletins being produced that FlowFiles are being auto-terminated via the success relationship. Have you verified that nothing is written to hbase? Have you verified all your hbase configurations are correct? The flow above indicates the issue is more of a throughput problem. The SplitJSON processor is producing large batches of output FlowFiles at one time (that is how that processor works). Back pressure is then applied on the connection between SplitJSON and PutHbaseJson processors (connection highlights red to indicate back pressure is being applied). SplitJSON will not be allowed to execute again until that back pressure drops below the configured threshold. The PutHBaseJson processor continues execute against FlowFiles in the connection queue. Have you tried the following to see if you see an improvement in throughput of the PutHbaseJson processor: 1. Increase configured batch size. The default is 25. Keep in mind that the content of every FlowFile in a batch is held in heap memory. Setting this value too high can lead to Out Of Memory (OOM) condition in NiFi's JVM. So increment slowly. 2. Increase configured concurrent tasks. Default is 1. One concurrent task allows this processor to execute on 1 batch at a time. Until that batch put is complete the next batch cannot be processed. Adding additional concurrent tasks will allow this processor to execute on multiple batches of FlowFiles at the same time. Keep in mind that you system does have a limited number fo cores. Setting this value too high can lead to your CPU becoming saturated. The max threads all processors on canvas must share is set via "Controller Settings" found in the global menu in the upper right corner of the NiFi UI. General rule of thumb is to set the "Maximum Timer Driven Thread Count" to a value equal to 2 to 4 times the number of cores on your NiFi server. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Soumya Ghosh 1. The DistibuteMapCacheServer (DMC server) has not direct linkage to a dataflow built in NiFi. Thus when creating a template of your flow from NiFi for the purpose of generating your MiNiFi yaml file, the DMC Server will not be included in the template. Only the DMC client service will be included. I have not tried to manually add the DMC server to the yaml file, but is likely possible. MiNiFi does not have a rest-api where you can send command to add additional components like you can do it NiFi. 2. The only way to host a DMC server is via a NIFi instance. There is not way to execute a NAR file. 3. A major limitation to using the DMC server is lack of HA. If the NiFi instance hosting the DMC server crashes you lost all your cached data (assuming crash is not recoverable. The recommended path is to configure you ListFile processor to use one of the other available external cache server that offer HA capability. These alternate cache service are setup independent of NiFi or MiNiFi and will offer you what you need to support using Entity tracking in your ListFile processor running on MiNiFi. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Ramesh Reddy When you build a dataflow on your NiFi UI's canvas and then save it as a template via "Create Template", all selected components (including all current configuration minus any passwords) will be saved into a template. The template is a just an xml snippet of that current configuration of those components which is kept in the NiFi's flow.xml.gz file. From the the "Templates" UI found under NiFi's global menu (upper right corner of NiFi UI) you can select any template and download that xml snippet to disk. Those snippets can then be uploaded to another NiFi for use there. The actual components on the canvas are in no way linked directly to any created template. So when you make changes to the components on the canvas, those changes are not pushed to any template you already created. You can create a new template anytime you want after making changes. I believe what you really want to be using is the NiFi-Registry service. This is a separate install from NiFi. NiFi can be configured to use the NiFi-Registry service to "Version Control" dataflows you build within the NiFi UI's canvas. Once a NiFi registry has been configured for your NiFi to use, you will have the option to right click on a NiFi Process Group (PG) and select "Start version control" from the version found in the presented context menu that is displayed. Once a PG is placed under version control, any changes made within the PG will trigger the PG to display an icon in upper left corner: informing you that local changes have been made. Right clicking on the PG again will allow you to commit those local changes as a new "version" of that dataflow in the NiFi-Registry. Other NiFi installs can be configured to use that same NiFi-Registry which allows those other NiFi instances to load a version controlled PG on to the canvas on that NiFi as well. A version controlled PG where the local flow matches teh current version in teh NiFi_Registry will display this icon in upper left corner of the PG: If a new version of a version controlled PG is pushed to the NiFi-Registry, any other NiFi instances using that version controlled flow will be notified that a newer version is available in the NiFi-Registry. User can then upgrade their PG to teh newer version. Thank you, Matt ... View more

@c i I am not clear from your description what your flow is doing. - What does the original FlowFile content look like before these replaceText processors? - How are the replaceText processors configured? The description you have of how the mergeContent processor works is not accurate. The MergeContent processor will merge a bin when any ONE of the following as occurred: 1. A given bin meets the configured minimum configured values for "number of entries" and "group size" (Bin-Packing Algorithm Merge Strategy) 2. A given bin contains all fragments of a fragmented FlowFile (Defragment Merge Strategy) 3. A bin has reached the configured "max bin age". Bin age starts when first FlowFile is allocated to a bin. The max bing age property exists to keep bins from staying around forever when the conditions outlined in 1 and 2 above are never met. 4. Processor runs out of bins. If FlowFiles have been allocated to each available bin already and a new FlowFile does not meet criteria to be addd to one of these existing bins, the oldest bin will be merged in order to free a bin to use for this new FlowFile. (This typically occurs when using the Correlation Attribute Name property and the source FlowFiles have more unique attribute values than bins allocated). The MergeContent processor will not always wait the configured max bin age before merging. Also keep in mind that with a NiFi cluster, each NiFi node is running the MergeContent processor independently of the other nodes and can only merge FlowFiles on the same node. So while an inbound connection queue to the MergeContent processor may show 2 queued FlowFiles, each of those FlowFiles may exists on different nodes and thus would not be merged together in to one FlowFile. Consider the "Max bin age" as the max latency you are willing to allow in your flow. So that a bin is merged even if it does not meet min criteria in the allotted time. So even if you set this to 30 minutes, a bin that meets the min criteria in 2 mins will get merged at 2 minutes. More details about your overall dataflow is probably needed to offer alternative suggestions here. Thank you, Matt ... View more

@Nick Stantzos You can ignore the unexpected coloring rendered by the NiFi Expression language editor window. The java regex replace should work just using "# DOG CAT BIRD" or "\# DOG CAT BIRD". Thank you, Matt ... View more

@Julian Iglesias - You can accomplish above using a function chain in your NiFi Expression Language (EL) statement. Rather then using the getDelimitedField() EL function: ${log_source:getDelimitedField(3,'\')} - You can successfully do this using the substringAfter() function twice to strip away what is before "dir2" and the substringBefore() function to strip away everything after "dir2": ${log_source:substringAfter('\\'):substringAfter('\\'):substringBefore('\\')} - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more