Hello @girish6   Your first EL statement would work as well once you correct the single quote marks around the Z.  Many outside text editors end up messing up the single quotes, so the NiFi EL ends up being incorrect when you copy/paste. NiFi's EL editor built in to the processor property text blocks can help identify when the EL format is wrong. In above you can see your NiFi EL statement as copied and pasted on line 3. You'll notice the color coding indicates an issue with the function's input parameters. Line 1 has the properly formatted version of that same NiFi EL statement. Be careful when using copy paste from some external editors. Thank you, Matt ... View more

Hello @girish6,   NiFi Expression language does have a function that will increment a number, but it does not reset back to zero (unless you restart NiFi) and is incremented globally across call calls to that function.     So while this function will give each FlowFile a unique filename per NiFi node, there is no guarantee that they will be sequential or will start at 0 for new source zip file you extract.  If this works for your use case the NiFi Expression language statement would look like this in the UpdateAttribute processor: If you are just looking for filename uniqueness, a better solution may be to instead append the FlowFile's uniquely assigned UUID to the filename instead as follows:   Tips:  1. There is an expression language guide in the embedded help of your NiFi installation found by clicking on the global menu icon in upper right corner. 2. Within a NiFi processor property that support NiFi EL you can display a list of functions with details by typing "${" (signifies start of an EL statement) and then hold "Control" while clicking space bar.  This also works if you are mid statement and type ":"then "control" + click spacebar. Examples: file-${   (then cntrl+spacebar) ${uuid:  (then cntrl+spacebar) Hope this helps, Matt ... View more

Hello David,   The ReplaceTextWithMapping processor has been around for some time now and its design is rather simplistic in its current implementation. It is designed to match a simple regex that will map one group of non-whitespace characters to another group of characters (can have white space and back references).   in your example you are matching on a a set of characters that contains whitespaces.   If you were to add a replaceText processor to your flow to replace all the whitespace with a non-whitespace character like "-", then it will work. sample ReplaceText processor configuration: Your mapping file would of course then be based off this change: account-sing-up asu account-sign-in asi account-sign-out aso   the resulting mapping would then work:   Thank you, Matt   ... View more

@sho aa Based on your first screenshot provided, it appears the putHbaseJson processor is processing FlowFiles. The stats indicate that ~783,000 FlowFiles had been pulled from the inbound connection queue in the last 5 minutes. The PutHbaseJson processor also shows that it has an active thread (1) at the time of the screenshot. It appears you are auto-terminating both the success and failure outbound relationships on your PutHbaseJson processor. If FlowFiles were auto-terminating on the Failire relationship, you should be seeing a Red square (bulletin) which your screenshot does not show. The bulletin reports an error log was produced. So I can only assume if you are not seeing bulletins being produced that FlowFiles are being auto-terminated via the success relationship. Have you verified that nothing is written to hbase? Have you verified all your hbase configurations are correct? The flow above indicates the issue is more of a throughput problem. The SplitJSON processor is producing large batches of output FlowFiles at one time (that is how that processor works). Back pressure is then applied on the connection between SplitJSON and PutHbaseJson processors (connection highlights red to indicate back pressure is being applied). SplitJSON will not be allowed to execute again until that back pressure drops below the configured threshold. The PutHBaseJson processor continues execute against FlowFiles in the connection queue. Have you tried the following to see if you see an improvement in throughput of the PutHbaseJson processor: 1. Increase configured batch size. The default is 25. Keep in mind that the content of every FlowFile in a batch is held in heap memory. Setting this value too high can lead to Out Of Memory (OOM) condition in NiFi's JVM. So increment slowly. 2. Increase configured concurrent tasks. Default is 1. One concurrent task allows this processor to execute on 1 batch at a time. Until that batch put is complete the next batch cannot be processed. Adding additional concurrent tasks will allow this processor to execute on multiple batches of FlowFiles at the same time. Keep in mind that you system does have a limited number fo cores. Setting this value too high can lead to your CPU becoming saturated. The max threads all processors on canvas must share is set via "Controller Settings" found in the global menu in the upper right corner of the NiFi UI. General rule of thumb is to set the "Maximum Timer Driven Thread Count" to a value equal to 2 to 4 times the number of cores on your NiFi server. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Soumya Ghosh 1. The DistibuteMapCacheServer (DMC server) has not direct linkage to a dataflow built in NiFi. Thus when creating a template of your flow from NiFi for the purpose of generating your MiNiFi yaml file, the DMC Server will not be included in the template. Only the DMC client service will be included. I have not tried to manually add the DMC server to the yaml file, but is likely possible. MiNiFi does not have a rest-api where you can send command to add additional components like you can do it NiFi. 2. The only way to host a DMC server is via a NIFi instance. There is not way to execute a NAR file. 3. A major limitation to using the DMC server is lack of HA. If the NiFi instance hosting the DMC server crashes you lost all your cached data (assuming crash is not recoverable. The recommended path is to configure you ListFile processor to use one of the other available external cache server that offer HA capability. These alternate cache service are setup independent of NiFi or MiNiFi and will offer you what you need to support using Entity tracking in your ListFile processor running on MiNiFi. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Ramesh Reddy When you build a dataflow on your NiFi UI's canvas and then save it as a template via "Create Template", all selected components (including all current configuration minus any passwords) will be saved into a template. The template is a just an xml snippet of that current configuration of those components which is kept in the NiFi's flow.xml.gz file. From the the "Templates" UI found under NiFi's global menu (upper right corner of NiFi UI) you can select any template and download that xml snippet to disk. Those snippets can then be uploaded to another NiFi for use there. The actual components on the canvas are in no way linked directly to any created template. So when you make changes to the components on the canvas, those changes are not pushed to any template you already created. You can create a new template anytime you want after making changes. I believe what you really want to be using is the NiFi-Registry service. This is a separate install from NiFi. NiFi can be configured to use the NiFi-Registry service to "Version Control" dataflows you build within the NiFi UI's canvas. Once a NiFi registry has been configured for your NiFi to use, you will have the option to right click on a NiFi Process Group (PG) and select "Start version control" from the version found in the presented context menu that is displayed. Once a PG is placed under version control, any changes made within the PG will trigger the PG to display an icon in upper left corner: informing you that local changes have been made. Right clicking on the PG again will allow you to commit those local changes as a new "version" of that dataflow in the NiFi-Registry. Other NiFi installs can be configured to use that same NiFi-Registry which allows those other NiFi instances to load a version controlled PG on to the canvas on that NiFi as well. A version controlled PG where the local flow matches teh current version in teh NiFi_Registry will display this icon in upper left corner of the PG: If a new version of a version controlled PG is pushed to the NiFi-Registry, any other NiFi instances using that version controlled flow will be notified that a newer version is available in the NiFi-Registry. User can then upgrade their PG to teh newer version. Thank you, Matt ... View more