@anton123  I am still not completely clear on your use, but correct me if below is not accurate: 1. You fetch a single large file. 2. That file is unpacked in to many smaller files. 3. Each of these smaller files are converted in to SQL and inserted via the putSQL processor. 4. You then have unrelated downstream processing you don't want to start until all files produced by the unpackContent processor have been successfully processed by the putSQL processor. Correct? If so, the following exampe use case for the NiFi Wait and Notify processor is probably what you are looking to implement for this use case: https://pierrevillard.com/2018/06/27/nifi-workflow-monitoring-wait-notify-pattern-with-split-and-merge/ If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@zIfo  The TLS exception "unable to find valid certification path to requested target" is telling you that there is a  lack of trust in the handshake.  This means that the complete trustchain needed to establish trust is missing from the truststore. This is not an issue with the NiFi InvokeHTTP processor. From command line you could try using openssl to get the public certificates for the trusts chain from the target URL. (note that not all endpoints will return complete trust chain.   openssl s_client -connect <FQDN>:<port> -showcerts     The server hello in response to this command will have one too many public certs. each cert will have format of below example:   -----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBX MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UE CxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYx OTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoT GUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIx MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63 ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwS iV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351k KSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZ DrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zk j5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5 cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esW CruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499 iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35Ei Eua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbap sZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b 9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAP BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAf BgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIw JQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUH MAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6Al oCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAy MAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIF AwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9 NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9 WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw 9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy +qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvi d0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8= -----END CERTIFICATE-----    You can copy each (including the begin and end certificate lines) and place it in different <name>.pem files which you can then import each <name>.pem in to your existing truststore. A complete trust chain consists of all the public cert from signer of hosts cert to the self signed root CA public cert. If that signer cert is self-signed (meaning owner and signer have same DN), then it is considered the root CA.  If they are not the same, then another public cert exists in the chain.  A complete trust chain means you have all the public certs from the one that signed the target FQDN all he way to the root CA (owner and issuer the same DN). If the output of the openssl does not contain all the public certs in the trust chain, you'll need to get the missing public certs from the source.  That source could be the company hosting the server or it could be a public certificate authority (Digicert for example).  You would need to go to to those sources to obtain the CA certs (often published on their website (example: https://www.digicert.com/kb/digicert-root-certificates.htm). Another option that may work for root CAs and some intermediate CAs is using java's cacerts file bundle with every java release which contains the public certs for many public authorities. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

Thanks Matt ... seeing and fixing all directories ownership was key to start again NiFi ... ... View more

Thank you, man   I can now see hive processors     ... View more

@sathish3389  Routing based on a sensitive value is an unusual use case.  I'd love to hear more about this use case.  Ultimately the RouteOnAttribute processor expects a boolean NiFi Expression Language Statement.  So you want to have a sensitive parameter value that is evaluated against something else (another attribute on the inbound FlowFile) and if true route to a new relationship.  Is what you are comparing this sensitive parameter value against also sensitive?  If so, how are you protecting it as Attributes on FlowFiles are not sensitive and stored in plaintext. The ability to use Sensitive Parameters in dynamic properties (non password specific component properties) was added via https://issues.apache.org/jira/browse/NIFI-9957 in Apache NiFi 1.17.0.  While this change created the foundation for such dynamic Property support for sensitive parameters, individual components need to be updated to utilize this new capability. As you can imagine with well over 300+ components available to NiFi, this is a huge undertaking.  So what i see in the apache community are changes based on specific use case requests. I'd recommend creating an Apache NiFi Jira detailing your use case and working with the Apache Community to adopt that use case change to the RouteOnAttribute processor to support dynamic property support for Sensitive parameters. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@MaarufB  You must have a lot of logging enabled that you expect multiple 10MB app.log files per minute. Was NiFi ever rolling files? Check your NiFi app.log for any Out of Memory (OOM) exceptions.  Does not matter what class is throwing the OOM(s), once the NiFi process is having memory issues, it impacts everything within that service.  If this is the case, you'll need to make changes to your dataflow(s) or increase the NiFi heap memory. Secondly, check to make sure you have sufficient file handles for your NiFi process user. For example; - If your NiFi service is owned by the "nifi" user, make sure the open file limit is set to a very large value for this user (999999).    A restart of the NiFi service before the change to file handles will be applied. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Onkar_Gagre  The Max Timer Driven Thread pool setting is applied to each node individually.   NiFi nodes configured as a cluster ate expected to be running on same hardware configuration.  The guidance of 2 to 4 times the number of cores as starting point is based on the cores of a single node in the cluster and not based off the cumulative cores across all NiFi cluster nodes.  You can only reduce wait time as you reduce load on the CPU.  In most cases, threads given out to most NiFi processors execute for only milliseconds.  But some processors operating against the content can take several seconds or much longer depending on function of the processor and/or size of the content. When the CPU is saturated these threads will take even longer to complete as the CPU is giving time to each active thread.  Knowing the only 8 threads at a time per node can actually execute concurrently, a thread only gets a short duration of time before giving some to another.  The pauses in between are the CPU wait time as thread queued up wait for their turns to execute.  So reducing the max Timer Driven Thread count (requires restart to reduction to be applied) would reduce maximum threads sent to CPU concurrently which would reduce CPU wait time.  Of course the means less concurrency in NiFi.  Sometimes you can reduce CPU through different flow designs, which is a much bigger discussion than can be handle efficiently via the community forums.  Other times, your dataflow simply needs more CPU to handle the volumes and rates you are looking to achieve.   CPU and Disk I/O are the biggest causes of slowed data processing. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Vinylal, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.   ... View more

Hi @MattWho , I did lot of investigation on the same but was not sure merge record can do that. Thanks a lot for your help. ... View more