@Chetan_mn  I loaded up the latest NiFi-2.0.0-M4 (milestone 4 release). Loaded up my flow definition used in my NiFi 1.23 version. All seems to work fine sending headers with mix case and seeing the correct attributes created with those mix case headers on FlowFile generated by HandleHTTPRequest processor. InvokeHTTP: You'll see two custom headers (displayName and outerID) added above as dynamic properties. HandleHTTPRequest processor: When I "List Queue" on the connection containing the "Success" relationship from HandleHTTPRequest processor and "view details" the queued FlowFile, the FlowFile attributes look correct. Are you saying you see different? Try using NiFi 2.0.0-M4 (latest) to see if experience is same. At what point in your dataflow are you checking the validating the FlowFile Attributes. Is your custom script maybe handling them wrong? I am not seeing an issue in the HandleHTTPRequest processor with regards to HTTP Header handling. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Chetan_mn  While I do not have an install currently of Technical Preview NiFi 2.0 milestone2 release, I used a NiFi 1.18 to build a simple dataflow using HandleHTTPRequest. I then setup an invokeHTTP processor to send a message to  to that api endpoint using the PATCH http method. I also include a couple custom headers: displayName=Display1 outerID=123456aBcD When I inspected the received FlowFile from HandleHTTPRequest, I see the FlowFile attributes created from the headers look correct: I suggest you try using an InvokeHTTP processor to test your HandleHTTPRequest processor in Apache NiFi 2.0.0-M2  to make sure your issue is not the result of some external manipulation of the headers before they are received by the HandleHTTPRequest processor. The headers are just create as FlowFile Attribute property names.  I am curious how the all lowercase of these property names are impacting your dataflow? Are the values for your headers being modified? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Techie123  When you say "run it manually", does that mean you simply start the processor and allow it to run continuously or are you right click on the processor and selecting "run once"? How do you have the "scheduling" configured for the processor?  I assume you are trying to use cron? Thank you, Matt ... View more

@Leo3103  This exception might be caused by a missing nar file in your MiNiFi. MiNiFi does not include all the NiFi nars. https://github.com/apache/nifi/blob/main/minifi/minifi-docs/src/main/markdown/minifi-java-agent-quick-start.md A quick look through your flow definition I saw you were using the ExecuteSQL processor. While ExecuteSQL is part of one of the included nars, it does require another nar that is not included (this is noted in the above linked quick start guide). You'll need to add the "nifi-dbcp-service-nar" from your NiFi distribution to your MiNiFi's lib directory. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

I think you issue may be with using the SingleUserAuthorizer and Single user login provider.  These out of the box providers were built so that NiFi could be HTTPS enabled securely out of the box.  They are not designed to support clustering, nor are they suitable for production NiFi. You'll want to configure your NiFi cluster to use a production ready authorizer (managed authorizer) and user authentication method other then single user so you can have granular access controls per user/team.  Most common is the ldap-provider. The documentation provides examples for authorizer.xml setup: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-ldap-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-kerberos-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap-based-usersgroups-referencing-user-dn ETC.... You cluster is most likely not forming completely due to node to node authentication and authorization issue resulting from using the single user authorizer.   In a NiFi cluster the node identities (derived from clientAuth certificates in the Mutual TLS exchange) need to be authorized against some NiFi policies like "Proxy user requests". Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@hegdemahendra  The FlowFile connection back pressure thresholds are soft limits.  Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again. So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute.  During a single execution it is consuming more events then the threshold settings.   What is the batch size set to in your ConsumeAzureEventHub processor? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@Rohit1997jio  You have topic A with your source messages.  You have two consumer groups each pulling all the messages from Topic A.  While both these dataflows consume all the same messages, each may fail on a different messages with the InvokeHTTP execution.   You are then writing the FlowFiles that failed invokeHTTP to another topic R which both consumer groups can consume from. So both consumer groups will get a copy of any message written to the topic.  Your dataflow is working exactly as designed.  You must keep your retry logic independent of one another.   I also don't understand the overhead of ingesting the same messages twice in your NiFi. Why not have have a single ConsumeKafka ingesting the messages from the topic and then routing the success relationship from the ConsumeKafka twice (once to InvokeHTTP A and once to InvokeHTTP B)?   Why publish failed or retry FlowFile messages to an external topic R just so they can be consumed back in to your NiFi?  It would be more efficient to just keep them in NiFi and create a retry loop on each InvokeHTTP.  NiFi even offers retry handling directly on the relationships with in the processor configuration. If you must write the message out to just one topic R, you'll need to append something to the message that indicates what InvokeHTTP (A or B) failure or retry resulted in it being written to Topic R.  Then have a single Retry dataflow that consume from Topic R, extracts that A or B identifier from message so that it can be routed to the correct invokeHTTP.  Just seems like a lot of unnecessary overhead. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@Chetan_mn  Details are very beneficial in getting assistance in the community. Why different NiFi instance would produce two different FlowFile attribute contents, makes no sense to me.  1. Are these two instance of NiFi just  2 nodes in the same NiFi  cluster? 2. What version(s) of NiFi are being used? 3. What processor is being used to your http requests?  I am assuming the HandleHttpRequest processor. 4. How is the HTTP processor configured? 5. Can you share a sample http request? 6. What is the source generating the request?  Is it an automated process? Thank you, Matt ... View more

@AlokKumar  The idea is that after renaming your FlowFile, you use the putFile processor to write that file to your archive directories on each of the NiFi instances.  That ends that dataflow. You then start a new dataflow that consists of only the GetFile processor with its success relationship terminated and the minimum File Age property set to "30 days".  Start this processor and it will continuously check the target archive directory for any files where the last modified timestamp has exceeded 30 days.  Those files older then 30 days will be consumed and removed from archive assuming property "Keep Source File"  is set to "false".  Since the success "relationship" is set to auto-terminate, The FlowFile produced is terminated. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more