Member since
07-30-2019
3472
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 278 | 06-03-2026 06:06 PM | |
| 547 | 05-06-2026 09:16 AM | |
| 1092 | 05-04-2026 05:20 AM | |
| 616 | 05-01-2026 10:15 AM | |
| 722 | 03-23-2026 05:44 AM |
01-11-2024
01:22 PM
@JamesZhang Certainly a challenging issue you have here. The shared output all points to good certificates, but gets you no closer to why the mutualTLS exchange between your two Nifi nodes is no yielding a successful mutual TLS handshake. I guess I would start by looking at the configuration of NiFi on both nodes to make sure configurations in the nifi.properties files on both nodes match. Verify that both nodes NiFi's are using same Java version. You may need to look at the network traffic between both nodes as well. Is there some device (load balancer, firewall, etc) between those nodes on the network that may be interfering with the certificate exchange. Matt
... View more
01-11-2024
01:16 PM
1 Kudo
@glad1 Based on what you shared, you may be able to accomplish what you are trying to do using a couple additional processors and a Child Process Group that utilizes "Process Group FlowFile Concurrency" settings. So your use case involves: For each 1 FlowFile output by your ExecuteSQLRecord processor you want to do ALL the following before the next FlowFile produced by ExecuteSQL record is processed: 1. Split the FlowFIle into X number of FlowFiles using SplitJson. 2. Modify each one of the produced Split FlowFiles using UpdateRecord. 3. Write all those modified FlowFiles ot another Database using PutDatabaseRecord 4. ExecuteSQL only once to update Record that all splits were processed. Then repeat above for next produce FlowFile If this is correct, here is what you might want to try: 1. create a Process Group that you will insert in teh middle of this flow as shown in the following image: 2. Configure that Process group as follows: Important properties to set here are: - Process Group FlowFile Concurrency = Single FlowFile Per Node - Process Group Outbound Policy = Batch Output What this does is allow only one FlowFile (per node in multi-node NiFi) to enter this PG at any given time. Inside this Process group you will have handle the processing of this FlowFile (split, update, putDB). The outbound policy will not release any of the Produced Splits from teh PG until all are queued at the output port. You'll notice I added one additional optional processor ModifyBytes to your dataflow (configured with "Remove all Content = true). This will zero out the content on the FlowFiles after they were written using the PutDatabaseRecord processor. Then those FlowFiles with no content now are sent to connection feeding output port where they are held until all splits produced are queued. They will then all be released at once from the PG to the next new processor MergeContent. The MergeContent processor will merge all those FlowFiles into a single FlowFile that feeds your ExecuteSQL (UpdateRecordStatus toP) processor. Now you have a single notification for the original FlowFile that was split and processed. Additionally you have created separation between each source FlowFile processed at start of dataflow. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-10-2024
01:23 PM
@Sartha I think i may now be understanding the confusion here. MiNiFi does not alter in any way the functionality of the components (NiFi processor, controller services, etc). They all function exactly as they do in NiFi. As I understand it now, your use case is to tail a file located on a different server from where NiFi or MiNiFi is installed. Correct? Neither NiFi or MiNiFi can tail a log file located on another server. The TailFile processor can only tail files local to the same server on which NiFi or MiNiFi is running. So MiNiFi would need to be installed on the server where the log file exists in order to tail that file. MiNiFi installed on that server would then be able to transfer the generated FlowFile over the network to your NiFi running on your local machine. MiNiFi is essentially NiFi without the UI (few other differences). Either NiFi or MiNiFi would need to be installed in that server. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-10-2024
01:09 PM
1 Kudo
@Alexy Sorry about the confusion, it shoudl have said hours instead of days in my response: 2. Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 hours unless TotalSizeCap is reached before 10 hours of logs are created. So you example here: nifi-app_2024-01-09_11.0.log
nifi-app_2024-01-09_11.1.log
nifi-app_2024-01-09_11.2.log
nifi-app_2024-01-09_11.3.log Implies you are using the pattern: <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> This also means you are using "maxFileSize" property as well that controls when each incremental hour log is rolled. Without a "TotalSizeCap" property set, logs will be retained for 2 hours or until you run out of disk space since there is no boundary on the number of incremental logs that may roll within each of those 2 hours. nifi-app_2024-01-09_11.0.log
nifi-app_2024-01-09_11.1.log
nifi-app_2024-01-09_11.2.log
nifi-app_2024-01-09_11.3.log
nifi-app_2024-01-09_11.4.log
nifi-app_2024-01-09_11.5.log
...
nifi-app_2024-01-09_12.0.log
nifi-app_2024-01-01_12.1.log
nifi-app_2024-01-09_12.2.log
nifi-app_2024-01-09_12.3.log
nifi-app_2024-01-09_12.4.log
... If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-10-2024
12:45 PM
@FrankHaha Have you tried using the "Infer Schema" Schema Access Strategy in the JsonTreeReader 1.24.0 controller services instead of fetching schema from AvroSchemaRegistry? Another option would be to use the ExtractRecordSchema 1.24.0 processor along with JsonTreeReader 1.24.0 controller services configured with "Infer Schema" Schema Access Strategy" to output the schema into a FlowFile Attribute "avro.schema". You can then get the produced schema from that FlowFile Attribute to add to your AvroSchemaRegsitry for future use. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-09-2024
07:25 AM
@Sartha I am very confused on your use case. here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log. If you are just trying to tail the nifi-app.log, what do you need MiNiFi for here... Let step back and just get a clear understanding of yoru use case. What are you trying to accomplish? Remove NiFi from the equation and define your use case. i.e - I have server1 and server2, I have some service running on server 2 writing log files. I want to read those logs as they are being written to and send that log output to server 1. etc... As more detail the better. As far as processors go, there is detailed documentation for each one. https://nifi.apache.org/docs/nifi-docs/ Understanding which NiFi components you need to use starts with a detailed use case. I am still not clear on your use case and where MiNiFi fist in to it if you are trying to tail the nifi-app.log. Tailing the nifi-app.log implies you have nifi runing on the server where the nifi-app.log is being written to, thus you could just use NiFi to tail its own nifi-app.log. But why do this? What are you trying to capture from nifi-app.log? What are planning to so with this log output once you tail it? Thank you, Matt
... View more
01-09-2024
07:03 AM
@JamesZhang The logs shared indicate a TLS exchange issue. Have you looked at the output of openssl to see what your running NiFi responds with: openssl s_client -connect runtime-0.runtime-statefulset.default.svc.cluster.local:443 -showcerts and openssl s_client -connect runtime-1.runtime-statefulset.default.svc.cluster.local:443 -showcerts
... View more
01-09-2024
06:06 AM
@Alexy 1. TotalSizeCap is across all log files, it is not based on anything else. So with this set depending on number of incremental (%i) 50 MB logs are created each hour, you may ned up retaining less then 10 hours of logs. 2. Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created. Logback is not developed by NiFi and has some oddities to it. It works well when using both maxHistory and TotalSizeCap, but you may see odd behavior when only using MaxHistory. When only MaxHistory is used, Logback does not look back further than configured maxHistory. So while a continually running service works great, things get weird when the service is down for a period of time the extends beyond the maxHistory. Lets say you have logs for last 10 hours (hours 1 -10) already and you bring down NiFi for a couple hours. So now you have logs that go back 12 hours (hours 3-12)when you restart the service. Logback will ignore the hour 11 and hour 12 logs since it is only looking to purge the hour 10 logs when the hour rolls over. So a few hours later you would still see the logs for hour 11 and 12 lingering around with a continuous 10 hour block of logs logback is still tracking on. The TotalSizeCap property however would make sure those old logs get deleted. This odd behavior to me was also brought up with logback community, but ultimately community decided to keep this behavior. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-09-2024
05:47 AM
@glad1 The first question is how do you know when all the FlowFiles have been processed through the PutDatabaseRecord processor? Here I see FlowFiles queued in every connection. So before you can tackle your question, need to understand more about your complete dataflow. 1. How does the flow of data begin? 2. Does it come in as one FlowFile that is split? 3. Does it come in batches? How do you know when a complete batch is ingested? 4. Is ingestion based on cron? 5. Does the ExecuteSQL use content of FlowFile Attributes from the incoming FlowFiles when it is executed? Thanks, Matt
... View more
01-08-2024
12:30 PM
@JamesZhang What is the verbose output for your configured truststore? Does it contain the TrustedCertEntry for your Certificate Authority (CA) that signed the PrivateKey in your keystore? The keystore you shared has: DNSName: runtime-0.runtime-statefulset.default.svc.cluster.local The log output you shared is failing on the mutualTLS handshake with another node in your NiFi cluster when the request to get current user is replicated to all nodes in your NiFi cluster. runtime-1.runtime-statefulset.default.svc.cluster.local All inter node communication require successful mutualTLS exchanges. Did you create a separate certificate for the other node? Is it signed by same CA? If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more