@Rohit Bhattacharya I am not clear on when you say " but data is not streaming in the output file". Processors in NiFi perform a task against a FlowFile. The replaceText performs a replaceText task against the content of a single FlowFile from an inbound connection queue. It might be helpful here if you provide an example input FlowFile's content and what you want the output FlowFile's content to then look like. Thank you, Matt ... View more

@Jake As newer version of Apache NiFi are released, many components (processors, controller services, reporting tasks) get bug fixes and enhancements/new features added to them. What you are seeing here is that the version of Apache NiFi where this dataflow template was originally generated was an older version which did not have these three new properties. These properties are not custom properties but rather new standard properties. When you upload a template build on Apache NiFi 1.5 for example to Apache NiFi 1.7 all the components in that template will use get upgraded to the Apache NiFi 1.7 versions. At times this may render some processors invalid (when new required properties (in bold) are added and do not have default values) and at other times the default values assigned to new properties may cause issues (as you are experiencing here). The three new properties associated to lines you shared above are: You will need to set these properties to values needed for your SQL needs. The documentation imbedded in each release of NiFi will be specific to that installed version. That documentation can be found by clicking on the global menu icon in the upper right corner of the NiFi UI and the clicking on "Help" in the presented menu. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Sebastian Carroll There are other things that also exist in heap memory space within the NiFi JVM: Component Status History: NiFi will store status history data points for all processors on the NiFi canvas (including those that are stopped). You can see this stored status history by right clicking on a component and selecting "view status history". Each component has numerous stats for which these data points are retained. All these component status points are stored in heap memory. The number of points per each stat that is held in heap is controlled in the nifi.properties file: nifi.components.status.repository.buffer.size --> Specifies the buffer size for the Component Status Repository. The default value is 1440. nifi.components.status.snapshot.frequency --> This value indicates how often to present a snapshot of the components' status history. The default value is 1 min. so on every restart of NiFi these stats are gone since they are in heap only. Then over the course of default 24 hours (1440 minutes in 24 hours) the heap usage grows. You can reduce the heap usage by this status history by adjusting the above properties. (take snapshots less frequently, perhaps every 5 minutes. Reduce number of data points retained from 1440 to 380 or lower.) Templates: All uploaded templates (whether they are instantiated to canvas or not) are held in heap memory. you can reduce heap memory usage by deleting uploaded templates you will no longer be instantiating to canvas. Dataflow: Your entire flow is held in heap memory. The more components you have on the canvas the large heap footprint. Queued FlowFiles: Even if no processors are running, the FlowFile attributes for FlowFiles loaded in to each connection between processors are held in heap memory. (There is a swap threshold configurable in nifi.properties file which triggers a connection to start swapping to disk if number fo queued FlowFiles exceeds the configured swap threshold) Thank you, Matt ... View more

@satish v - Screenshot may help us understand the questions you are asking better. 1. What file are you trying to save as an .xml? How are you performing this action? Are you using a NiFi dataflow you created on the canvas within the NiFi UI? NiFi does not have a "file".. "menu/tool" bar. 2. NiFi was not designed to be a job tool. Processor components are generally started and left that way. A "running" processor will execute its code based on the configured "scheduling". Option exist in the scheduling tab of a processor to execute based on a timer or a cron. There is a third event driven strategy option which should never be used (experimental). 3. What video tutorial are you referring to? link? 4. When you say "processing", what does that mean? What is your use case? 5. NiFi has over 300 unique components (processor controller service, reporting tasks, etc...) which allow user of NiFi to do many things. These components come from many different contributors to this open source software. There isn't going to be tutorials for every component. It is best to provide a detailed use case for which you are looking for help with. Highly recommend you create a new question in he forum for each unique use case. different community members will have input on different use cases and you are less likely to get the assistance you need if everything is lumped in to one forum question. - Keep in mind that Apache NIFi provides rather robust component documentation embedded right inside the application. It can be found by clicking on the global menu icon in the upper right corner (looks like three horizontal lines) and then selecting "help". You can also go directly to a specific processors documentation by right clicking on a processor and select "view usage" from the context menu that appears. - Welcome to the Apache NiFi. - Thank you, Matt ... View more

@Raj Negi The Process Group variables are not dynamic. Anytime the variable is changed it requires a restart of any component that references it. That is because those components read in that variable value when they are started in a lot of cases. - Your first dataflow is responsible executing an InvokeHTTP processor to retrieve the token needed for all the different dataflows within the same Process Group. You may consider adding a PutDistributedMapCache processor in that flow to write that retrieved token out to a cache server. In each of your other dataflows you could have a FetchDistributedMapCache processor that pulls that token from the cache server so it is is added to every FlowFile. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Guilherme Varela - NiFi is always go to favor data duplication over data loss. In this case you have your ScrollElasticSearchHttp processor executing on Primary node only. When the primary node changes, the primary node only processors are stopped on old primary node and started on new primary node. This can result in duplication because state is only updated at completion of the the currently executing thread. When the processor is stopped on the old primary node, any active thread is not killed. The processor goes in to a "stopping" status and is only completely stopped once all active threads have been completed or terminated. Meanwhile on the newly elected primary node it has been asked to start and thus reads from the same cluster state which has not been updated yet by the still executing thread on the old primary node. - Typically the election of a new primary node is the result of the old primary node becoming disconnected from the cluster or not communicating with ZK resulting in a new election of another node. Either way there is some connective issue (which may be temporary and go unnoticed by the user) and NiFi can only assume that that node may have crashed. The new primary node has no way of knowing if that old primary node is still executing on that processor. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

Or are you looking for a unique ID that is assigned to every FlowFile that is traverses a specific set of processors (dataflow) on the NiFi canvas. That way you can track which FlowFiles were processed by which dataflow? If so, rather then using the UUID() NiFi Expression Language (EL) function , why not just have an updateAttribute processor in each on of these unique dataflows set a static value on each FlowFile? for exampe: sessionID = dataflow1 ... View more

@Identity One You may want to look in to using the detectDuplicate processor and a cache server to check for duplicate serial numbers. Perhaps a flow that extracts the serial number from an incoming data and place that serial number in an attribute on the FlowFile. Then that FlowFile is passed to a detectDuplicate processor which checks the serial number via the newly created FlowFile attributes against a distributed Cache server. If serial number is already found, FlowFile is deleted as duplicate, if it is a new serial number, it is added to the cache and FlowFile routed further along in dataflow. If you still wanted to maintain a file on disk of all the serial numbers, you could take the serial number from the non-duplicate FlowFiles an append them to the file on disk just as you are now. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@rohit chavan Absolutely, Simply configure your listenHTTP processor with a SSLContextService NiFi Controller Service. The controller service with be used to provide the keystore and truststore necessary to facilitate the two-way TLS/SSL handshake from the connecting client(s). When configured with a SSL Context Service, the processor's jetty server will only accept TLS 1.2 connections. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Sai aditya - Let me make sure I understand what you are doing first... 1. You are using an UpdateAttribute processor to add a new attribute to each FlowFile that pass through that processor. correct? 2. This attribute on each FlowFile has an attribute name of "fnnn" and some value assigned to that attribute. correct? 3. This FlowFile then flows to an invokeHTTP processor where you need to add the value associated to that attribute to the URL. correct? Every NiFi Expression Language (EL) statement starts with "${" and then ends with "}". You will notice in above screenshot I have modified what you provided with a proper EL of "${fnnn}". - The ReplaceText processor is used to modify the content of a FlowFile. Are you looking to modify the FlowFile content? - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Micah Pearce There are numerous ways to accomplish the same result in NiFi. The use of the ${retry.counter:replaceNull('0'):plus(1)} NiFi Expression language statement is to create an attribute with value 1 if attribute "retry.counter" does not exist on the FlowFile; otherwise, the existing value currently assigned to existing attribute "retry.counter" is returned and incremented by 1. - If we just set it to one, each time the same FlowFile passes through this loop it would get set to 1 and would never increment. - Thank you, Matt ... View more

@Vivek Thakur You can put whatever value you like in there. - for example: 48 hours 30 days 90 days etc.... - Setting a value of say 90 days here does not guarantee that you will even have 1 day of provenance history. The amount of provenance history retained is going to be capped by another Provenance configuration and rate at which your dataflow(s) are producing provenance events. - nifi.provenance.repository.max.storage.size The maximum amount of data provenance information to store at a time. The default value is 1 GB. The Data Provenance capability can consume a great deal of storage space because so much data is kept. For production environments, values of 1-2 TB or more is not uncommon. - The more processors in use and the higher the number of FlowFile being processed through these processors will equate to higher number of produced provenance events. - If you are looking to store large amounts of provenance events, you may want to consider adjusting the defaults on the below setting as well to have better provenance query performance: - nifi.provenance.repository.index.shard.size The repository uses Apache Lucene to performing indexing and searching capabilities. This value indicates how large a Lucene Index should become before the Repository starts writing to a new Index. Large values for the shard size will result in more Java heap usage when searching the Provenance Repository but should provide better performance. The default value is 500 MB. However, this is due to the fact that defaults are tuned for very small environments where most users begin to use NiFi. For production environments, it is advisable to change this value to 4 to 8 GB. Once all Provenance Events in the index have been aged off from the "event files," the index will be destroyed as well. Note: this value should be smaller than (no more than half of) the nifi.provenance.repository.max.storage.size property. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link ... View more

@Aaron R The only method you can use to systematically adjust the configured run Schedule of a NiFi processor is through the use of the NiFi rest API. You would achieve this through a series of invokeHTTP processors. - 1. Use invokeHTTP processor to stop another processor. 2. Use invokeHTTP processor to change "Run schedule" configuration on a processor. 3. Use InvokeHTTP processor to start processor again. (Note that the processor will immediately execute when you start it and then not execute again for newly configured "run schedule" duration.) - You would then need to repeat above process for all three of your sensor inputs. Since I do not know the full details of your use case, it is not clear if this solution will really meet you needs. However, I can tell that based on how NiFi was developed, this is the only method for adjusting this configuration without user manual intervention. - You may just be better off building a flow that sends a FlowFile to each of the sensors invokeHTTP processor to trigger their execution. The InvokeHTTP processor does accept an inbound connection. When setup this way the invokeHTTP processor configured "run schedule" controls how often the processor checks that inbound connection for a FlowFile. When a FlowFile exists it will execute. - When a FlowFile is generated (by your "Sensor monitor controller" for example), that FlowFile will be timestamped. First you would need to split that incoming JSON into three different FlowFiles (one for each sensor). You could build a flow that parses the various sensor setting in to NiFi FlowFile Attributes and then create a if/then loop to check when the FlowFile timestamp + sensor value is equal to or less than current time. When that loop exits you route the FlowFile to appropriate invokeHTTP processor to trigger it to collect the sensor data. - Now becomes the question of what do you want to do with "Original" FlowFile out of the invokeHTTP. You can discard it and expect a new sensor monitor input each time or you can reset its trigger time and loop it back to the start. You would also need to in this case build in a method in your flow to kick this looping FlowFile out once a new sensor input comes in. (perhaps using the DistributedMapCache to store a value you increment each time a new timer comes in and then have your looping FlowFiles fetch this value to see if it has changed and kick FlowFile out if it has). - Hope these ideas help you with your design decisions. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Derek Calderon Unfortunately, a rolling restart will not work here. When you shutdown only one node another node in the cluster would get elected as cluster coordinator and still retain list of nodes known to be part of cluster. By shutting down all nodes you essential wipe the slate clean of what nodes are in cluster. On start nodes will come up, components (processors, controller services, reporting tasks. etc..) will return to last known state (running, stopped, etc..), FlowFiles will be loaded back in to last reported connection queue, and processing will continue. - One of the nodes will get elected as the cluster coordinator by Zookeeper and then other nodes will learn from ZK who was elected and start sending heartbeats directly to that elected cluster coordinator to join the cluster. As node join they will be added to list of connected nodes. - The dead node that never checks in will no longer be in the list. - No need to stop components before stopping NiFi. They will return to last known state on start. - Thank you, Matt ... View more