@bhadraka Check out this flow definition example for how to prepare and test JSON ahead of an invokeHttp post.   Flow Def File: https://raw.githubusercontent.com/cldr-steven-matison/NiFi-Templates/main/InvokeHttp_Demo_2.json     As far as the post itself goes.  I highly recommend that you test API calls with something like Postman so that you can clearly test what headers and such are required so that the API call works.  Only then try to make it work with InvokeHttp duplicating all headers.     Figuring out how to get the call to work and how to configure InvokeHTTP at same time is not fun. TIP:  in Invoke HTTP click the + to add key value pairs corresponding to your headers or values you need to pass.     ... View more

@Juanes I believe you just need to resolve the missing dependency.    Check out this solution: > pip3 install python-setuptools > yum install impala-shell https://community.cloudera.com/t5/Support-Questions/How-to-install-impala-shell-on-RHEL-8-3-to-communicate-with/m-p/313665   ... View more

@Manimaran Wait/Notify should be able to do what you are describing in the deeper dialouge.   Based on your current flow complexity,  I would recommend that you build a new sample to learn how wait/notify behavior works in nifi.   This is very difficult setup, so having a learning flow w/o context of the existing functionality is highly optimal.   Then you can refactor your sql flow accordingly, without having to figure out how to get wait/notify working during that refactor. ... View more

@DTM In that case you would need to use the DatabaseConnectionPool and jdbc to aws postgres.  This will require permissions to allow nifi network to speak to RDS endpoint.  If you cant use DBCP,  you will have to put something between the RDS and nifi.  For example nifi could use invokeHTTP to send/post data to an ec2 instance with some kind of API that can do the connectivity. ... View more

@DTM You are correct, to use an AWS credentials in nifi, you need to use the Controller Service.   This controller service is then referenced by processors as a drop down menu for AWS Credentials Provider Service.  If one does not exist in the drop down you can chose to create it.   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.21.0/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html   An example of one such processor is GetDynamoDB but there are many.  Just search AWS in the processor search box to find all aws related processors. ... View more

@drewski7   This blog is a great place to start:   https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/   That said, some recommendations:   Recommend 3 nodes. Use 32 or 64gb ram.  Set min ram 16, max 32, let nifi/operating system leverage other 32gb. Add more cores and tune Active Thread Count accordingly Be careful which processors are Primary Only and which processors are not.    Do not over loadbalance queues, load balance at top of flow, let nifi distribute work load naturally after that. Tune Processor Concurrency and Run Schedule.  Be sure to understand how each work. With a good setup tuned as above, have a plan to identify when time is appropriate to scale horizontally (add more nodes).   Here are some more docs that get specific into sizing:   https://docs.cloudera.com/cfm/2.1.1/nifi-sizing/topics/cfm-sizing-recommendations.html https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.2/nifi-configuration-best-practices/content/configuration-best-practices.html       ... View more

@Ray82 Yes, you can achieve this with UpdateRecord. You will need to provide record reader/writer with schema of your upstream and downstream.  Then in UpdateRecord you explicitly add properties (+) for each record value you want to update versus using a SQL statement like QueryRecord.     Here are some useful community posts on this topic:   https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267   https://community.cloudera.com/t5/Support-Questions/NiFi-UpdateRecord-processor-is-not-updating-JSON-path/m-p/186256   ... View more