Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 149 | 12-17-2025 05:55 AM | |
| 210 | 12-15-2025 01:29 PM | |
| 143 | 12-15-2025 06:50 AM | |
| 261 | 12-05-2025 08:25 AM | |
| 423 | 12-03-2025 10:21 AM |
12-13-2019
06:46 AM
@varun_rathinam The only method offered by the MergeContent Processor to force a bin to merge when it has not reached the configured minimum set values is the "Max Bin Age" property. How long does it take to accumulate 500 FlowFiles in a bin for merge? Set your Max Bin age to a value higher than that time duration. Now to avoid merging FlowFile from day 1 and day 2 (scenario where day 2 files start showing up at your MergeContent processor before end of day 1 max bin age time has been reached), I suggest using the "Correlation Attribute Name" to make sure only FlowFiles from the same day are placed in the same bin. This would require you to extract the date/day from your FlowFiles somehow (maybe by HDFS dir path, or filename, or some file metadata if these exist). Hope this helps, Matt
... View more
12-13-2019
06:37 AM
1 Kudo
@Biswa NiFi's Expression Language (EL) will not support wildcard in attribute subject names. I suggest you pull height and width in to a set of standardized attribute names: Use the UpdateAttribute processor advanced UI to create some rules. Fore example: if "PNG-IHDR.Image Height" attribute exists, create new attribute "image height" with value from "PNG-IHDR.Image Height" Do the same for the other 3 unique attribute names, thus resulting in a static attribute subject names for height and width for use later in your flow for comparison/routing decisions. Hope this helps, Matt
... View more
12-13-2019
06:27 AM
@girish6 By xml file, I am assuming you are referring to the flow.xml.gz file? NiFi will only read the flow.xml.gz file during NiFi startup. While NiFi is running the flow.xml resides completely in JVM heap memory and updates made via the UI or rest-api calls are pushed to flow.xml.gz only. It can be very dangerous manually editing the flow.xml.gz file and is not recommended. Your queued FlowFiles are tightly coupled to specific connection uuids. While the NiFI UI/rest-api would block you from removing a connection which has queued FlowFiles, the flow.xml.gz would not offer such protections. This could result in data loss if someone where to manually edit the flow.xml.gz file and restart NiFi. Also keep in mind that in a multi-node NiFi cluster each node runs with its own copy of the flow.xml.gz loaded in to memory and the UI and rest-api calls are replicated to all nodes to make sure these files stay synced across your cluster. Hope this helps, Matt
... View more
12-12-2019
11:57 AM
@girish6 Anything you can do through the NiFi UI can also be accomplished via the NiFi rest-api. Within NiFi you can go to the global menu in the upper right corner and then select "Help". On the left side of the Help UI you will see and option in the list for the NiFi rest-api docs for the release you are running. However, it is easier to get use to NiFi rest-api through the developer tools available in your browser. You can open the developer tools while you are in the NiFi UI and see the rest-api calls as they are made when you perform an action on the GUI (like adding a processor component). You can even right click on the rest-api call in the developers tool window and copy as curl to get the exact rest-api call you could make from a command line prompt. Hope this helps you, Matt
... View more
12-12-2019
11:43 AM
@kam1 Processors are not coded to expose the exceptions to the FlowFile. Typically how this handled is by Using the SiteToSIteBulletinReportingTask. Anytime a component throws an exception the processor will display a bulletin in the upper right corner and that same bulletin will be sent to the bulletin board. The same exception along with stack trace (if there is one) is also logged to the nifi-app.log. The S2SBulletinReportingTasks allows you to capture all produced bulletins and send them via S2S to another NiFi or even back to same NiFi to be processed as FlowFiles for notification and alerting reasons through a dataflow. Writing exceptions out to FlowFile attributes would be problematic for following reasons: 1. FlowFile attributes exist in NiFi heap memory and exceptions can be large. 2. A FlowFile may pass multiple processor which produce bulletin/exception. Is expectation we overwrite previous exception with new exception? Or create a new FlowFile attribute based on processor uuid for each exception encounter during life of FlowFile? This leads back to point 1 issue. It is better to handle exceptions as content within their own FlowFiles via the reporting task mentioned above. Hope this helps, Matt
... View more
12-12-2019
11:14 AM
@fowler7878 I am not sure what you want to accomplish is doable via your design. The GetHDFS processor will either find a file to get or it will produce no output FlowFiles. So when it finally does produce a FlowFile because it found a file, the path attribute I am sure will be something recent depending on how often you schedule the GetHDFS processor to run. If you are only trying to notify someone if there has been a lapse of 20 days without getting a single file from HDFS, you may want to instead look at the monitorActivity processor. You would add this after your GetHDFS processor so it monitors for FlowFiles coming from your GetHDFS. You can configure it with a 20 day threshold. It can also notify when activity resumes after a lapse was reported. Hope this helps you, Matt
... View more
12-11-2019
11:12 AM
@Dewashish NiFi 1.10 introduced a new parameters feature that allows you to define parameters that can be used in any component property. https://issues.apache.org/jira/browse/NIFI-6276 After reading the user guide for this new feature, it does not look like it can be used in NiFi controller services or reporting tasks. https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters I recommend that you open an Apache NiFi Jira requesting that this particular property be recoded as a sensitive property. Hope this helps, Matt
... View more
12-11-2019
11:06 AM
@fowler7878 No problem. At this point you have extracted the the count successfully and used updateAttribute processor to subtract 31 from it. You do not need to write it back to the content to use it in putEmail processor. The majority of the putEmail processor configuration properties support NiFi Expression Language also. You can retrieve the value of your FlowFiiles attribute with this simple NiFi EL statement: ${COUNTFILES} So you can use above anywhere in the string you configure in the the "Subject" and/or "Message" configuration properties. Thanks, Matt If you found my answer help you to a solution, please don't forget to mark it as the accepted solution.
... View more
12-11-2019
10:30 AM
@xpelive Just want to make sure a couple configurations. 1. The port you are using in the URL (8180) is the "nifi.web.http.port=" and not the "nifi.remote.input.socket.port=" port configured in your nifi.properties file. 2. When you start your NiFi instance, you will see a line output in the nifi-app.log when startup is complete that says the NiFi Ui is available at the following URLs: Is "localhost:8180" listed in one of those URLs? If not, you need to use one of the URLs listed in your Remote Process Group. It might be that localhost is bound to a different NIC on your server than what NiFi is listening on. 3. If you have or will ever have more than 1 NiFi node in a NiFi cluster, using "localhost" is not going to work. Your NiFi nodes should be configured to use the actual hostnames fo the server on which the service is running. That hostname should be configured in both these properties: --- nifi.remote.input.host= --- nifi.web.http.host= 4. Since your NiFi is not secured, make sure that your nifi.properties file is not configured to be secure: nifi.remote.input.secure=false. Null exceptions are particularly difficult to diagnose because a null basically means there was no error handling coded to handle to particular exception. Thus we need o look at all angles of the configuration to see where the issue may reside.
... View more
12-11-2019
10:13 AM
1 Kudo
@craignelson7007 Since you are starting with JSON content which you are extracting bits of into FlowFileAttributes just to write them back to json, have you considered skipping this completely and just using the NiFi JoltTransformJson processor to modify the existing json content instead? Here is a article with more details on using the JoltTransformJson processor: https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350 Otherwise: NiFi stores all FlowFile attributes as strings. The json you have written out in a FlowFile Attribute is going to be stored as a string in the attribute key vckey_list = ["test value"] The AttributesToJson processor builds a flat json from those attribute strings, so it is going to wrap the entire json in quotes and escape the embedded quotes within that string with a backslash. You could use a couple replaceText processors after your AttributesToJson processor to remove the unwanted quotes and backslashes from the newly created content. Here is another reference around this processor limitation: http://apache-nifi.1125220.n5.nabble.com/Nesting-and-AttributesToJSON-td3575.html Hope this helps, Matt
... View more