Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 270 | 05-06-2026 09:16 AM | |
| 451 | 05-04-2026 05:20 AM | |
| 331 | 05-01-2026 10:15 AM | |
| 514 | 03-23-2026 05:44 AM | |
| 388 | 02-18-2026 09:59 AM |
01-18-2024
09:55 AM
@Alexy MaxHistory is relative to timedBased Policy date pattern: in the example shown in this thread you mentioned: https://mkyong.com/logging/logback-xml-example/ The full rollingPolicy example is: <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<!-- each archived file, size max 10MB -->
<maxFileSize>10MB</maxFileSize>
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
<totalSizeCap>20GB</totalSizeCap>
<!-- 60 days to keep -->
<maxHistory>60</maxHistory>
</rollingPolicy> Within the fileNamePattern we see the configured date pattern "%d{yyyy-MM-dd}" which is based on days. So, maxHistory of 60 means in this specific example to retain "60" days. Now if that date pattern instead was "%d{yyyy-MM-dd_HH}" which is based on hours instead, the same maxHistory=60 would now mean 60 hours of logs. But since this policy also archives on size, the fileNamePattern also includes "%i" which roles the log incrementally each time it reaches the configured maxFileSize= 10MB. So it will keep 60 days of logs; however, each of those 60 days could have 1 too many rolled logs depending on how much logging is output. app.2024-01-18.1.log.gz app.2024-01-18.2.log.gz app.2024-01-18.3.log.gz app.2024-01-18.4.log.gz app.2024-01-18.5.log.gz ... So if your application decides to unexpectedly start producing large amounts of log output, you could quickly consume all your disk space with incremental logs. This is where the TotalSizeCap setting comes into the picture. It's job is to say regardless of number of logs you want to keep (60 days in this specific configuration example), start deleting old archives when total log archives exceeds this configured size cap. maxHistory is relative to the configured date pattern and is always days. So comment in above exact example is correct. it is not about number of log files and since pattern is based in days, it is correct. A better generic comment is: <!-- Control the maximum number of log archive files kept and asynchronously delete older files --> since it makes no specific relative to the dat pattern being used. It simply states that the maxHistory value correlates to number of log archives that are retained. What that equates (minutes, hours, days, etc) to depends on the user defined data pattern. I modified the example in the article you liked that I created with above comment instead of days even though I called it out later in my article for clarity. In my opinion, TotalSizeCap is something you should use anytime you use the SizeAndTimeBasedRollingPolicy. That is because maxHistory setting does not take into account the incrementally (%i) created log files. So using above example where it is creating daily logs (%d{yyyy-MM-dd}) the maxHistory= 60 says to keep 60 days of logs. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-18-2024
09:17 AM
1 Kudo
@MPHSpeed Working with the actual data instead of sample data I build, i would recommend making these two changes: 1. In extractText processor change "Enable Unix Lines Mode" to true. 2. In each dynamic property in RouteOnAttribute, change "equals" function to "contains" function. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-18-2024
06:54 AM
@Sartha localhost means same host on which the service is running, so you can't use localhost in a postHTTP processor running on MiNiFi (host B) and expect it to sent FlowFiles to ListenHTTP running on another host (host A). The postHTTP is going to transmit FlowFiles to another host, so why would you configure postHTTP running on your MiNiFi Host with the MiNiFi IP and port? in the PostHTTP url used is server hostname and port number because MiNiFi is running on this server Maybe I am just misunderstanding what you are saying in above statement. It also does not appear that the URL is correct. It appears to be missing the "Base path" (default contentListener) as described in the processors documentation. To help you out, I build this flow so I can share screenshots of how it is configured for you: Here is PostHTTP configuration which would be used on MiNiFi (host B): Here is configuration of ListenHTTP used on the NiFi instance (host A) that will receive FlowFiles from above PostHTTP on your MiNiFi (host B): In my configuration I used port 9933, but you can use any unused non privileged port available on your NiFi instance (host A). I am also still confused by your screenshots with regards to below: Image 1 of the tailFile processor. This the processor that is feeding the PostHTTP processor on the MiNiFi instance installed on (host B), correct? You configured it to tail a nifi-app.log (which is not configured with the absolute path) which is generated by NiFi (host A). As I mentioned before, the tailFile processor can not tail a file located on a different host form where the tailFile is executing). Image 3 showing dataflow. Why is ListenHTTP success relationship connected to PostHTTP? If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-18-2024
06:14 AM
@Alexy I looked at the default logback.xml bundled with the latest NiFi 2.0 releases and see which is much better description of what it does: <!-- Control the maximum number of log archive files kept and asynchronously delete older files -->
<maxHistory>30</maxHistory> Control maximum number does not mean it is exactly the configured number being retained since maxHistory applies the TimeBasedRolling policy. older versions of NiFi have following which is misleading: <!-- keep 30 log files worth of history -->
<maxHistory>30</maxHistory> I know a lot of committers and they are not going to go back to older releases and update it there. NiFi went to Logback 1.3 as of Apache NiFi release 1.20 as part of: https://issues.apache.org/jira/browse/NIFI-10580 later upgraded to: 1.3.8 ---> Apache NiFi 1.23.1 and 1.24.0 NIFI-11881 1.3.14 --> Apache NiFI 1.25 and 2.0.0-M2 NIFI-12462 If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-17-2024
01:37 PM
@MPHSpeed Rather than using RouteText processor which routes individual lines of a text file, you could use RouteOnContent processor that routes the entire FlowFile whose content matches to a dynamic relationship. What i would do is extract the data type TDRS 3, or AMSC 1 or SKYNET 4C, etc to a FlowFile attribute using ExtractText processor and then you have that type associated with the FlowFile through your entire flow making it easy to do things like merge FlowFiles all of same type together (MergeContent with "Correlation Attribute Name"), route FlowFiles of a specific type using RouteOnAttribute, etc... Then you also have options using the many Record based processors if you can define a schema for your data that defines your record as those three lines. Example: SplitText (splits relationship) ---> ExtractTEXT: ExtractText (Matched relationship) --> RouteOnAttribute RouteOnAttribute with above configuration will have three dynamically created relationships for the data types you want to keep. connect each to the unique dataflow path for processing that data type. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-17-2024
07:32 AM
@pratschavan Based on the exception shared it is telling you that the ingested msg files you have do not follow RFC-2822 specification the extractEmailHeader controller service has a requirement. States that the particular MSG it tried to process was missing the sender. You may need to write your won customer reader for your formatted MSG files and without, unfortunately I will not be very helpful as it is outside my area of knowledge. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-17-2024
07:28 AM
@Alexy I have never used cleanHistoryOnStart and the docs are not super clear. Archive removal normally happens at time of log rotation. This is why older logs beyond window in which they should be retained linger around when application is down and then started up. I cam across this post that has some interesting responses: https://stackoverflow.com/questions/54680434/the-old-log-files-didnt-get-removed-when-using-logback-rollingfileappender Implied from that that post it is not removing the files because of your rolling policy %d{yyyy-MM-dd_HH} If what they are saying is correct, it will not remove your old logs because they are form a different day and your log rotation is based on hour within current day. I more commonly uses "totalSizeCap" setting a value here larger than what i expect to see retained in my desired history time window or even larger based on available disk space. This has worked for me to clean old stuff up so it is not around forever. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-17-2024
06:58 AM
@Sartha Your MiNiFi needs a simple dataflow that consists of a TailFile processor and then either a Remote Process Group (RPG) (recommended if your NiFi is a multi-node cluster, but can be used with single instance NiFi as well.) or a PostHTTP processor (Use if NiFi a single instance and not a cluster). Neither the RPG not the PostHTTP can be configured with a target URL with "localhost". Localhost would be the MiNiFi server. It needs to be the hostname of the server where your NiFi is running. Make sure if using RPG, you have configured the Site-to-Site properties in the nifi.properties file. Your NiFi would need a Remote Input Port (this is what your MiNiFi RPG will transmit FlowFiles to) or a ListenHTTP processor (if you used PostHTTP on your MiNiFi, this is what it can be configured to send to). The outbound connection from either of these components needs to feed whatever downstream processors you need to do within NiFi. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-17-2024
06:37 AM
1 Kudo
@glad1 No not necessary. I suggested becasue i was still unclear how often your initial ExecuteSQL was producing a source file. The PG makes it easy to throttle per source FLowFile processing so you would get one merged FlowFile for each produced FlowFile. Thanks, Matt
... View more
01-12-2024
07:00 AM
@manishg I am not clear on what you are trying to accomplish here. What is the use case? What is your NiFi version? What is your OS? NiFi does not have a "start.sh" script. Are you talking about the "nifi.sh" script. Perhaps there are just some important details I am missing here. also not sure why you would want to change the nifi..web.http.port configuration property in the nifi.properties file to a variable. These properties are all read during startup of NiFi and evaluating NiFi variables is does not happen during NiFi startup. Nor does NiFi support defining NiFi variables in the nifi.properties file. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more