Member since
10-09-2019
12
Posts
0
Kudos Received
0
Solutions
03-08-2020
11:01 PM
Thanks @MattWho
... View more
11-06-2019
05:32 AM
2 Kudos
@girish6 NiFi processor components are configured to execute based on a run schedule. There are two schedule driven strategies available (Cron Driven and Timer Driven). The Cron Driven scheduling strategy uses a user configured Quartz Cron to set how often the processor will execute. The Timer Driven scheduling strategy (most common strategy used) uses a user configured run schedule (default run schedule is 0 secs, which means run as often as system will allow). When a processor executes based on the configured scheduling strategy, it will do one of two things: 1. If the processor has one or more inbound connections, it will check if any of them have any queued FlowFiles. If none of the connections contain any queued FlowFiles, the processor will yield. The yield is intended to keep the processors with run schedule of 0 secs from simply constantly requesting CPU threads to check empty inbound connection queues. No matter the run schedule, a yielded processor will not execute until the yield has expired reducing CPU usage by that processor. 2. Some processor have no inbound connections. These processors will not yield, but continuously execute on the configured run schedule. You would not have any such processors in your PG2 since they will have upstream connections to components in PG1. So for "source" type processors like listSFTP, ListFile, GenerateFlowFIle, or any other processor that does not support an inbound/upstream connection, if the feed of data is not continuous, it is best to use the Cron Driven scheduling strategy or set a Timer Driven run schedule that is not the default 0 secs to reduce CPU usage. On the face of every processor is a state for Tasks/Time. The stat tells you how many threads reported as completed in the past 5 minutes and how much cumulative CPU time was used by all those completed threads. This allows you to see the impact a given processor is having on your CPU. Hope this helps explain cpu usage for you, Matt
... View more
11-05-2019
04:29 AM
1 Kudo
@girish6 The logic you are trying to accomplish is not possible with the existing scheduling strategies that are available to NiFi processor components? I have no idea which processor you are talking about here, but since you mention it is looking for FlowFiles, I can only assume it is a processor with an incoming connection from another component. What is your concern with just having it run all the time? What about using a different cron so that it runs every 5 mins during hour 3 each day? This allows it a full hour each day to look for those incoming FlowFiles. Thanks, Matt
... View more
10-21-2019
05:29 AM
@girish6 NiFi allows you to specify multiple variable registry files in the nifi.properties file. There is only the one property: nifi.variable.registry.properties This is a comma-separated list of file location paths for one or more custom property files. These files are only read by NiFi on startup, so any changes or additional files added to this property will not take affect until you restart NiFi. The variable registry file was added to NiFi before it became possible to set NiFi Variables directly within the NiFi UI on process groups. The advantage to the newer variables that can be set on process groups is that additional variables and changes to existing variables can be made without needing to restart NiFi. From the NiFi Ui you can see a list of all components that are currently using a set variable. Changing a variable via the UI also will handle stopping and restarting any component already referencing that variable. Bonus: Keep an eye out for "NiFi Parameters" coming soon. Unlike variables, parameters can be used in any NiFi component property field. Variables are only support in component properties that support NiFi Expression Language (EL). Parameters will also support sensitive properties (meaning they are able to store passwords in a secure encrypted format. Hope this helps, Matt
... View more
10-10-2019
10:15 AM
Hello @girish6 Your first EL statement would work as well once you correct the single quote marks around the Z. Many outside text editors end up messing up the single quotes, so the NiFi EL ends up being incorrect when you copy/paste. NiFi's EL editor built in to the processor property text blocks can help identify when the EL format is wrong. In above you can see your NiFi EL statement as copied and pasted on line 3. You'll notice the color coding indicates an issue with the function's input parameters. Line 1 has the properly formatted version of that same NiFi EL statement. Be careful when using copy paste from some external editors. Thank you, Matt
... View more
10-10-2019
09:58 AM
1 Kudo
Hello @girish6, NiFi Expression language does have a function that will increment a number, but it does not reset back to zero (unless you restart NiFi) and is incremented globally across call calls to that function. So while this function will give each FlowFile a unique filename per NiFi node, there is no guarantee that they will be sequential or will start at 0 for new source zip file you extract. If this works for your use case the NiFi Expression language statement would look like this in the UpdateAttribute processor: If you are just looking for filename uniqueness, a better solution may be to instead append the FlowFile's uniquely assigned UUID to the filename instead as follows: Tips: 1. There is an expression language guide in the embedded help of your NiFi installation found by clicking on the global menu icon in upper right corner. 2. Within a NiFi processor property that support NiFi EL you can display a list of functions with details by typing "${" (signifies start of an EL statement) and then hold "Control" while clicking space bar. This also works if you are mid statement and type ":"then "control" + click spacebar. Examples: file-${ (then cntrl+spacebar) ${uuid: (then cntrl+spacebar) Hope this helps, Matt
... View more