@Alexy  1. TotalSizeCap is across all log files, it is not based on anything else.  So with this set depending on number of incremental (%i) 50 MB logs are created each hour, you may ned up retaining less then 10 hours of logs. 2. Max History does not count incremental (%i) logs generated.  It is based on the date pattern used %d{yyyy-MM-dd_HH}.  So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created. Logback is not developed by NiFi and has some oddities to it.  It works well when using both maxHistory and TotalSizeCap, but you may see odd behavior when only using MaxHistory.  When only MaxHistory is used, Logback does not look back further than configured maxHistory.  So while a continually running service works great, things get weird when the service is down for a period of time the extends beyond the maxHistory. Lets say you have logs for last 10 hours (hours 1 -10) already and you bring down NiFi for a couple hours.  So now you have logs that go back 12 hours (hours 3-12)when you restart the service.  Logback will ignore the hour 11 and hour 12 logs since it is only looking to purge the hour 10 logs when the hour rolls over.  So a few hours later you would still see the logs for hour 11 and 12 lingering around with a continuous 10 hour block of logs logback is still tracking on.   The TotalSizeCap property however would make sure those old logs get deleted.  This odd behavior to me was also brought up with logback community, but ultimately community decided to keep this behavior. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@glad1  The first question is how do you know when all the FlowFiles have been processed through the PutDatabaseRecord processor?  Here I see FlowFiles queued in every connection.  So before you can tackle your question, need to understand more about your complete dataflow.  1. How does the flow of data begin?  2. Does it come in as one FlowFile that is split?  3. Does it come in batches?  How do you know when a complete batch is ingested? 4. Is ingestion based on cron? 5. Does the ExecuteSQL use content of FlowFile Attributes from the incoming FlowFiles when it is executed? Thanks, Matt ... View more

@JamesZhang  What is the verbose output for your configured truststore? Does it contain the TrustedCertEntry for your Certificate Authority (CA) that signed the PrivateKey in your keystore? The keystore you shared has: DNSName: runtime-0.runtime-statefulset.default.svc.cluster.local The log output you shared is failing on the mutualTLS handshake with another node in your NiFi cluster when the request to get current user is replicated to all nodes in your NiFi cluster. runtime-1.runtime-statefulset.default.svc.cluster.local All inter node communication require successful mutualTLS exchanges. Did you create a separate certificate for the other node?  Is it signed by same CA? If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@JamesZhang  Not sure what "cut the login" means in your response. When you access the NiFI URL, are you being redirected to the NiFi login window or do you encounter the unknown certificate exception immediately? Where did you get the certificates you are using? Did you add the Certificate Authority CA trust chain public certificates to the list of trusted authorities in the browser you are using to connect to NiFi? Thanks, Matt ... View more

@Sartha  Let me make sure I am clear on what you have built. MiNiFI: 1. You have deployed MiNiFi on a remote server.  You built a dataflow on a NiFi install consisting of a TailFile processor feeding a Remote ProcessGroup. That flow was converted in to a yml and used in your deployed MiNifi. 2. The TailFile has been configured to tail some log file(s) on the remote server.  I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log? 3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network.  I see you configured the RPG with an http url.  Is your NiFi install running over http or https?  Out-of-the-box apache NiFi runs secured over https.  So did you remove that security so it is running un-secure over http?  If not the RPG is not going to be able to connect to your NiFi.  If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi.   If MiNiFi is on a different server then your NiFi, "localhost" is not going to work.  If it is on same server, what is the point of using MiNiFi? NiFi: 1. Is your NiFi a  multi-node cluster setup or single standalone instance?  If it is a single standalone instance you would probably be better off building a different flow between your MiNiFi and NiFi. (MiNiFi: TailFile --> PostHTTP or InvokeHTTP.  NiFI: ListenHTTP --> other processors).  If it is a NiFi cluster stick with RPG as it will distribute FlowFiles across all nodes.  NOTE: With the postHTTP or InvokeHTTP to ListenHTTP dataflow, security is handled directly via the configuration of those processors, so it can be used without needing to care if MiNiFi and/or NiFi is secured or not. 2. If your NiFi is secure, what NiFi authorizer are you using? Out-of-the-box Nifi uses a Single-User authentication provider and Single-user-Authorizer.  These out of the providers do not support setting up additional clients/users authorizations so will not support RPGs.   So first you would to need to change to a different authorizer like FIle-authorizer.  Secured communication between MiNiFi RPG and NiFi's Remote Input Port will require authentication (handle via mutual TLS exchange)  and authorization (handled by NiFi authorizer).   There are several authorization you would need to setup for your MiNiFi client before sending FlowFiles securely to NiFi will be possible (retrieve site-to-site details and receive data via site-to-site). You did not share what you are seeing in the MiNiFi logs or NiFi logs when this dataflow is running to understand the issue(s) you may be having. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@JamesZhang  I feel there are a lot of details missing here that may help you get a better response. I see you added the "Apache NiFi" label, so assuming you are seeing this exception some how related to NiFi? javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown Above is telling you that you have some TLS exchange  issue related to some certificates somewhere.  I am assuming the verbose output you shared is for the keystore configured in your NiFi's nifi.properties file? With and TLS exchange there is a client and a server side of that exchange and the keystore and truststores on both side of that exchange along with the type of TLS exchange (TLS or MutualTLS) matters.   Initial questions; 1. Where are you seeing this exception? What action is being performed when the exception occurs? 2. What TLS exchange is failing as a result of it? Thank you, Matt ... View more

@arutkwccu  The Apache NiFi 2.0.0-M1 release notes have now been updated with a list of nars that have been moved to the Optional Build Profiles.   https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1 Thank you, Matt ... View more

@Elnozy  Other then you examining a serious of spaced out thread dumps from those two nodes during the long startups, it would be difficult to know the specific reason.  But you could look to see if these two nodes have a lot more queued data then the other two nodes.    Are their content, flowfile, and provenance repositories a lot larger then the other nodes? As fars as the numerous known startup improvements made over the years, there are no workarounds to them other then getting those improvements through upgrade. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@jerome69007  How was your nifi.service created? I am confused as to why it contains both the following lines: ExecStart=/appli/nifi/bin/nifi.sh start ExecStop=/appli/nifi/bin/nifi.sh stop I would expect the nifi.service to only contain the start command. This explains why on linux startup it is executing the start followed by the stop command. Typically setting NiFi as a service on your linux is performed in a few simple steps: 1. Execute "./nifi.sh install" command.  This will setup NiFi as a service allowing to use systemctl command like "systemctl start nifi", "systemctl status nifi", and "systemctl stop nifi".   2. execute "systemctl enable nifi" command to configure nifi service start on system bootup. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Elnozy  HDF 3.3 (Based off Apache NiFi 1.8) is almost 7 years old at this point in time.  CFM 2.1.6 is the latest Cloudera NiFi offering based off Apache NiFi 1.23+ version.   There have been many improvements to NiFi over the years including fixes that greatly improve NiFi startup time. Your best option is to upgrade to mitigate many security issues, bugs fixes, and get the many improvements including those that improve startup speed. The difference in startup times is most likely directly associated to the amount of FlowFiles queued on some nodes versus others.   But, here are just some examples of fixed issues related to slow startup: NIFI-9382 NIFI-9289 NIFI-7999 If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more