@plapla  A secured NiFi is secured over HTTPS which means a TLS exchange happens to secure the connection with NiFi.  A secured NiFi will always support a mutual TLS exchange. If no other methods of user authentication are configured, NiFI will "REQUIRE" a clientAuth certificate be presented in the TLS exchange with NiFi. When NiFi is configured with an additional user authentication method (for example, you have enabled the ldap-provider for user authentication), NiFi will "WANT" a clientAuth certificate in the TLS exchange.  If a clientAuth certificate is not provided in the TLS exchange/handshake, NiFi moves on to the next authentication method configured. The ldap-provider will require obtaining a user token as you saw that then needs to be included with all subsequent rest-api calls.  And you are correct that the token does expire.  That is why it is easier and a better option to use mutual TLS based authentication when doing automation like this.  The clientAuth certificate is simply included in every rest-api request and there is no token involved.   About the ClientAuth certificate...  The full DN from the certificate is what is used to identify the user.  The full DN is evaluated against any identity.mapping.pattern.xxx properties configured in the nifi.properties file.  If a configured pattern (java regex) matches against the DN, the identity.mapping.value.xxx and identity.mapping.transform.xxx is applied.  These identity mappings are often used to trim the CN value from the complete DN.  The resulting string after any mappings are applied is what is then used to look up authorizations for that client/user. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Sofia71  The HandleHTTPRequest processor establishes a generic endpoint, it has not idea what headers and in what format the content of those headers will be.  You client creates the request and decides what haaders and format of the header content.   I would recommend in yoru testing that you start the HandleHTTPRequest processor a keep the downstream processor stopped so that the incoming request becomes queued in the connection between the HandleHTTPRequest and the next downstream processor. You can then right click on the connection and list the flowfiles in the connection.  From the list you can view the details of the queude FlowFile which will aloo you to see the generated "http.headers.<some client derived string>" added as attributes to the FlowFile along with the values for those headers.    Using that information you can construct your validations. in the RouteOnAttribute processor. You'll need to verify the format of the authorization data pre encoding coming in the request header match exactly with the format of the authorization data you have put in the parameter context. You could also decode the authorization header contents to make sure it matches with what you constructed in your authorization parameter. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@plapla  This sounds like the putElasticSearchHTTP processor is working as designed.  It is putting to ElasticSearch over HTTP and ElasticSearch is successfully processing that request; however your ElasticSearch is not responding to the original http request before the timeout has occurred.  As a result, putElasticSearchHTTP has routed to failure. The question here is what are you doing with the failure relationship?  If you configured "retry" or looped the failure relationship via a connection back on the putElasticSearchHTTP processor, and the same FlowFile would be processed a second time. You may be able to solve this by simply increasing the configured "Response Timeout" configuration on the putElasticSearchHTTP processor.   But you may also want to look at the particular files that encounter this issue and see if their are any consistencies across them such as larger sizes, time of day, load on ElasticSearch at time, number of concurrent pending request on ElasticSearch side, network load, etc... If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@PriyankaMondal  Very simply... What @cotopaul responded with. One of the biggest definers of performance is your dataflow design itself.  Apache NiFi offers so many pluggable components for building out your dataflows and not all will perform the same.   While NiFi makes it easy to create dataflows, building the perfect highest performing dataflows can take some trial and error to get there.  I'd always recommend testing and modeling to understand the performance characteristics of the dataflow you built.  Identify and adjust where you see your bottlenecks.  Try different designs using different processors when possible.  Work with records instead of many small individual FlowFiles when possible for better performance. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Sofia71  The HandleHTTPRequest processor listens for incoming connection being sent to it from an external source and then relies on the HandleHTTPResponse processor to sent back the response to that incoming request.  So first question is how are you collecting this data?  Are you trying to fetch it?  If so, you should be using the InvokeHTTP processor instead. If the source is sending the data to your NiFi then you are using the correct processor. Doing any from of client based authentication would need to be need to be handled within your dataflow following the HandleHTTPRequest processor.  The processor itself will not do authorization and the only form of authentication it can do is mutualTLS based.  So for basic authorization you would need the user basic authentication presented in the request headers.  The HandleHTTPRequest processor will add those headers as attributes on the produced FlowFile.  You mention the authorization header username and password would be base64 encoded, so you could use NiFi Expression Language to via the UpdateAttribute processor and the base64decode function to decode them.  How you validate them is up to you after you have them. If they are LDAP based credentials, perhaps you could write a script you pass them to via one of the scripting processor to validate the username and password are correct?  If you want to keep it very basic, you could use an RouteOnAttribute processor that checks to see if username and password match what you say they should be and if they do, pass the FlowFile on downstream; otherwise, terminate the FlowFile there. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@oneofthemany  You would get better traction form the community by starting a new question since this question already has an accepted solution and has nothing to do with NiFi TLS-toolkit usage. Thank you, Matt ... View more