Member since
07-30-2019
3398
Posts
1621
Kudos Received
1001
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 484 | 11-05-2025 11:01 AM | |
| 374 | 11-05-2025 08:01 AM | |
| 596 | 11-04-2025 10:16 AM | |
| 734 | 10-20-2025 06:29 AM | |
| 874 | 10-10-2025 08:03 AM |
05-08-2019
08:51 PM
@3nomis MergeContent should be using Defragment. There is no default value for Max Bin Age, so not sure what you set there. If left blank, processor will wait for ever to merge a bin unless you run out of bins. Also make sure you adjust the object and size thresholds on the connections feeding the MergeContent processors so that they are large enough to accommodate the number of splits that need to be merged. Considering the size of the FlowFiles being merged, it may take time to merge all of them. as far as bins, try setting 21 of them. Thanks, Matt
... View more
05-02-2019
04:45 PM
Thanks Matt, I didn't realize that the colored text was not a comment in this case.
... View more
04-10-2019
06:55 PM
@Samar Aarkotti *** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion. It always best to leave your processor at default value for concurrent task unless there is a specific need to increment. Here is an article on this topic: https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor.html and another on "Run Duration": https://community.hortonworks.com/articles/221807/understanding-nifi-processors-run-duration-functio.html
... View more
04-04-2019
11:15 AM
@Matt Clarke Thanks a lot. You are genius!
... View more
03-25-2019
12:48 PM
@Aaron R The only method you can use to systematically adjust the configured run Schedule of a NiFi processor is through the use of the NiFi rest API. You would achieve this through a series of invokeHTTP processors. - 1. Use invokeHTTP processor to stop another processor. 2. Use invokeHTTP processor to change "Run schedule" configuration on a processor. 3. Use InvokeHTTP processor to start processor again. (Note that the processor will immediately execute when you start it and then not execute again for newly configured "run schedule" duration.) - You would then need to repeat above process for all three of your sensor inputs. Since I do not know the full details of your use case, it is not clear if this solution will really meet you needs. However, I can tell that based on how NiFi was developed, this is the only method for adjusting this configuration without user manual intervention. - You may just be better off building a flow that sends a FlowFile to each of the sensors invokeHTTP processor to trigger their execution. The InvokeHTTP processor does accept an inbound connection. When setup this way the invokeHTTP processor configured "run schedule" controls how often the processor checks that inbound connection for a FlowFile. When a FlowFile exists it will execute. - When a FlowFile is generated (by your "Sensor monitor controller" for example), that FlowFile will be timestamped. First you would need to split that incoming JSON into three different FlowFiles (one for each sensor). You could build a flow that parses the various sensor setting in to NiFi FlowFile Attributes and then create a if/then loop to check when the FlowFile timestamp + sensor value is equal to or less than current time. When that loop exits you route the FlowFile to appropriate invokeHTTP processor to trigger it to collect the sensor data. - Now becomes the question of what do you want to do with "Original" FlowFile out of the invokeHTTP. You can discard it and expect a new sensor monitor input each time or you can reset its trigger time and loop it back to the start. You would also need to in this case build in a method in your flow to kick this looping FlowFile out once a new sensor input comes in. (perhaps using the DistributedMapCache to store a value you increment each time a new timer comes in and then have your looping FlowFiles fetch this value to see if it has changed and kick FlowFile out if it has). - Hope these ideas help you with your design decisions. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.
... View more
03-21-2019
02:00 PM
@TRACEY JACKSON NiFi processors are configurable to run using a Timer Driven or Cron Driven scheduling strategy. This scheduling can uniquely configured per each processors configuration. Processors are then started and the operate based on the configured scheduler from that point forward. - The NiFi API can be used to perform any action a user can perform directly from the UI. The best way to learn the rest-api calls is to use the "Developer tools" available via most browsers. - While focused on tab where NiFi UI is open, launch the developers tools. Chrome Browser example: Then perform the action via the NiFi UI and you will see the "Network" call display in list. You can right click on that call and select "Copy as curl". Now you have an example of how to execute that same request via command line. - For more info in the NIFi rest-api, you can look in "help" found within the NiFi UI Global menu (upper right corner). The Rest Api documentation can be found in the "Developer" section at bottom of list on far left side of help UI. - The danger with trying to "run" dataflows in NiFi via command line is you may end up stopping processors in the dataflow chain that results in FlowFile being unprocessed and sitting on connection queues between processors. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.
... View more
03-22-2019
05:48 PM
1 Kudo
@Benjamin Bouret The invokeHTTP processor would require you to use a SSL context service when communicating with a secure (https) endpoint. The SSLContext service can be setup with only a truststore.jks if this is only a 1-way TLS connection that does not require client authentication. - You should be able to use openssl to get the complete public certificate chain from the target secured endpoint. From command line execute following command: Openssl s_client -connect <hostname>:<port> -showcerts - The return from this command will include one or more public certificates. each public certificate will start with and end with following: -----BEGIN CERTIFICATE-----
.....
-----END CERTIFICATE----- - Copy each certificate including the above two lines and write each to separate file with a .crt extension. for example: CA-1.crt - Then import each of these public certificates in to the truststore you want to use in your SSLContext service as follows: # keytool -import -alias <unique Alias name 1> -file CA-1.crt -keystore truststore.jks
# keytool -import -alias <unique Alias name 2> -file CA-2.crt -keystore truststore.jks
etc... - Make your your NiFi service user can read this file where ever you decide to place on each of your NiFi nodes. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.
... View more
03-20-2019
02:02 PM
@Matt Clarke,@matt burgess Exactly the second point is happening, each node is generating its own value incrementing from last value it has stored in its local state. So which processor or method should i use to generate an incremental batchid (batch1,batch2...so on) since update attribute is messing values when running on cluster. or is there any property by which updateattribute processors on all nodes can pickup each others's last state variable?..please suggest
... View more
05-08-2019
02:37 PM
@Mahadevan Ganesan Could possibilities come to mind here: 1. Based on your screenshot provided, you have configured a Max Bin Age of 2 minutes. The timer on a Bin starts when the first FlowFile has been allocated to that bin. It is possible that your bin is being merged at 2 minutes when it still has fewer than the desired 100 FlowFiles allocated to it yet. Try setting Max Bin Age to a higher value and see what the results are. 2. You have 10 allocated bins, but have possibly 10 or more unique schemas being used by your JsonTreeReader. Since only FlowFiles with like schemas will be allocated to the same bin. If when allocating a new FlowFile to a bin and that new FlowFile does match any existing bin, it will be allocated to a new bin. if no free bins exist, the oldest bin will get merged to free a bin. Verify all 100 source FlowFiles use exact same schema. Thanks, Matt
... View more