@Alexy  Sorry about the confusion, it shoudl have said hours instead of days in my response:   2. Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 hours unless TotalSizeCap is reached before 10 hours of logs are created.    So you example here:   nifi-app_2024-01-09_11.0.log nifi-app_2024-01-09_11.1.log nifi-app_2024-01-09_11.2.log nifi-app_2024-01-09_11.3.log   Implies you are using the pattern:   <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>   This also means you are using "maxFileSize" property as well that controls when each incremental hour log is rolled. Without a "TotalSizeCap" property set, logs will be retained for 2 hours or until you run out of disk space since there is no boundary on the number of incremental logs that may roll within each of those 2 hours.     nifi-app_2024-01-09_11.0.log nifi-app_2024-01-09_11.1.log nifi-app_2024-01-09_11.2.log nifi-app_2024-01-09_11.3.log nifi-app_2024-01-09_11.4.log nifi-app_2024-01-09_11.5.log ... nifi-app_2024-01-09_12.0.log nifi-app_2024-01-01_12.1.log nifi-app_2024-01-09_12.2.log nifi-app_2024-01-09_12.3.log nifi-app_2024-01-09_12.4.log ...     If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Sartha  I am very confused on your use case.     here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log.   If you are just trying to tail the nifi-app.log, what do you need MiNiFi for here... Let step back and just get a clear understanding of yoru use case.  What are you trying to accomplish?  Remove NiFi from the equation and define your use case. i.e - I have server1 and server2, I have some service running on server 2 writing log files.  I want to read those logs as they are being written to and send that log output to server 1. etc...  As more detail the better. As far as processors go, there is detailed documentation for each one. https://nifi.apache.org/docs/nifi-docs/ Understanding which NiFi components you need to use starts with a detailed use case. I am still not clear on your use case and where MiNiFi fist in to it if you are trying to tail the nifi-app.log.  Tailing the nifi-app.log implies you have nifi runing on the server where the nifi-app.log is being written to,  thus you could just use NiFi to tail its own nifi-app.log.  But why do this?  What are you trying to capture from nifi-app.log?  What are planning to so with this log output once you tail it? Thank you, Matt ... View more

@JamesZhang  The logs shared indicate a TLS exchange issue. Have you looked at the output of openssl to see what your running NiFi responds with: openssl s_client -connect runtime-0.runtime-statefulset.default.svc.cluster.local:443 -showcerts and openssl s_client -connect runtime-1.runtime-statefulset.default.svc.cluster.local:443 -showcerts   ... View more

@Alexy  1. TotalSizeCap is across all log files, it is not based on anything else.  So with this set depending on number of incremental (%i) 50 MB logs are created each hour, you may ned up retaining less then 10 hours of logs. 2. Max History does not count incremental (%i) logs generated.  It is based on the date pattern used %d{yyyy-MM-dd_HH}.  So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created. Logback is not developed by NiFi and has some oddities to it.  It works well when using both maxHistory and TotalSizeCap, but you may see odd behavior when only using MaxHistory.  When only MaxHistory is used, Logback does not look back further than configured maxHistory.  So while a continually running service works great, things get weird when the service is down for a period of time the extends beyond the maxHistory. Lets say you have logs for last 10 hours (hours 1 -10) already and you bring down NiFi for a couple hours.  So now you have logs that go back 12 hours (hours 3-12)when you restart the service.  Logback will ignore the hour 11 and hour 12 logs since it is only looking to purge the hour 10 logs when the hour rolls over.  So a few hours later you would still see the logs for hour 11 and 12 lingering around with a continuous 10 hour block of logs logback is still tracking on.   The TotalSizeCap property however would make sure those old logs get deleted.  This odd behavior to me was also brought up with logback community, but ultimately community decided to keep this behavior. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@glad1  The first question is how do you know when all the FlowFiles have been processed through the PutDatabaseRecord processor?  Here I see FlowFiles queued in every connection.  So before you can tackle your question, need to understand more about your complete dataflow.  1. How does the flow of data begin?  2. Does it come in as one FlowFile that is split?  3. Does it come in batches?  How do you know when a complete batch is ingested? 4. Is ingestion based on cron? 5. Does the ExecuteSQL use content of FlowFile Attributes from the incoming FlowFiles when it is executed? Thanks, Matt ... View more

@JamesZhang  What is the verbose output for your configured truststore? Does it contain the TrustedCertEntry for your Certificate Authority (CA) that signed the PrivateKey in your keystore? The keystore you shared has: DNSName: runtime-0.runtime-statefulset.default.svc.cluster.local The log output you shared is failing on the mutualTLS handshake with another node in your NiFi cluster when the request to get current user is replicated to all nodes in your NiFi cluster. runtime-1.runtime-statefulset.default.svc.cluster.local All inter node communication require successful mutualTLS exchanges. Did you create a separate certificate for the other node?  Is it signed by same CA? If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@JamesZhang  Not sure what "cut the login" means in your response. When you access the NiFI URL, are you being redirected to the NiFi login window or do you encounter the unknown certificate exception immediately? Where did you get the certificates you are using? Did you add the Certificate Authority CA trust chain public certificates to the list of trusted authorities in the browser you are using to connect to NiFi? Thanks, Matt ... View more

@Sartha  Let me make sure I am clear on what you have built. MiNiFI: 1. You have deployed MiNiFi on a remote server.  You built a dataflow on a NiFi install consisting of a TailFile processor feeding a Remote ProcessGroup. That flow was converted in to a yml and used in your deployed MiNifi. 2. The TailFile has been configured to tail some log file(s) on the remote server.  I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log? 3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network.  I see you configured the RPG with an http url.  Is your NiFi install running over http or https?  Out-of-the-box apache NiFi runs secured over https.  So did you remove that security so it is running un-secure over http?  If not the RPG is not going to be able to connect to your NiFi.  If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi.   If MiNiFi is on a different server then your NiFi, "localhost" is not going to work.  If it is on same server, what is the point of using MiNiFi? NiFi: 1. Is your NiFi a  multi-node cluster setup or single standalone instance?  If it is a single standalone instance you would probably be better off building a different flow between your MiNiFi and NiFi. (MiNiFi: TailFile --> PostHTTP or InvokeHTTP.  NiFI: ListenHTTP --> other processors).  If it is a NiFi cluster stick with RPG as it will distribute FlowFiles across all nodes.  NOTE: With the postHTTP or InvokeHTTP to ListenHTTP dataflow, security is handled directly via the configuration of those processors, so it can be used without needing to care if MiNiFi and/or NiFi is secured or not. 2. If your NiFi is secure, what NiFi authorizer are you using? Out-of-the-box Nifi uses a Single-User authentication provider and Single-user-Authorizer.  These out of the providers do not support setting up additional clients/users authorizations so will not support RPGs.   So first you would to need to change to a different authorizer like FIle-authorizer.  Secured communication between MiNiFi RPG and NiFi's Remote Input Port will require authentication (handle via mutual TLS exchange)  and authorization (handled by NiFi authorizer).   There are several authorization you would need to setup for your MiNiFi client before sending FlowFiles securely to NiFi will be possible (retrieve site-to-site details and receive data via site-to-site). You did not share what you are seeing in the MiNiFi logs or NiFi logs when this dataflow is running to understand the issue(s) you may be having. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@JamesZhang  I feel there are a lot of details missing here that may help you get a better response. I see you added the "Apache NiFi" label, so assuming you are seeing this exception some how related to NiFi? javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown Above is telling you that you have some TLS exchange  issue related to some certificates somewhere.  I am assuming the verbose output you shared is for the keystore configured in your NiFi's nifi.properties file? With and TLS exchange there is a client and a server side of that exchange and the keystore and truststores on both side of that exchange along with the type of TLS exchange (TLS or MutualTLS) matters.   Initial questions; 1. Where are you seeing this exception? What action is being performed when the exception occurs? 2. What TLS exchange is failing as a result of it? Thank you, Matt ... View more