@DTM In that case you would need to use the DatabaseConnectionPool and jdbc to aws postgres.  This will require permissions to allow nifi network to speak to RDS endpoint.  If you cant use DBCP,  you will have to put something between the RDS and nifi.  For example nifi could use invokeHTTP to send/post data to an ec2 instance with some kind of API that can do the connectivity. ... View more

@DTM You are correct, to use an AWS credentials in nifi, you need to use the Controller Service.   This controller service is then referenced by processors as a drop down menu for AWS Credentials Provider Service.  If one does not exist in the drop down you can chose to create it.   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.21.0/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html   An example of one such processor is GetDynamoDB but there are many.  Just search AWS in the processor search box to find all aws related processors. ... View more

@drewski7   This blog is a great place to start:   https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/   That said, some recommendations:   Recommend 3 nodes. Use 32 or 64gb ram.  Set min ram 16, max 32, let nifi/operating system leverage other 32gb. Add more cores and tune Active Thread Count accordingly Be careful which processors are Primary Only and which processors are not.    Do not over loadbalance queues, load balance at top of flow, let nifi distribute work load naturally after that. Tune Processor Concurrency and Run Schedule.  Be sure to understand how each work. With a good setup tuned as above, have a plan to identify when time is appropriate to scale horizontally (add more nodes).   Here are some more docs that get specific into sizing:   https://docs.cloudera.com/cfm/2.1.1/nifi-sizing/topics/cfm-sizing-recommendations.html https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.2/nifi-configuration-best-practices/content/configuration-best-practices.html       ... View more

@Ray82 Yes, you can achieve this with UpdateRecord. You will need to provide record reader/writer with schema of your upstream and downstream.  Then in UpdateRecord you explicitly add properties (+) for each record value you want to update versus using a SQL statement like QueryRecord.     Here are some useful community posts on this topic:   https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267   https://community.cloudera.com/t5/Support-Questions/NiFi-UpdateRecord-processor-is-not-updating-JSON-path/m-p/186256   ... View more

@MOUROU I recently built a nifi flow in version 1.21 that uses the NiFI API from within nifi, and it is NOT necessary to get access token.  From within nifi i am able to just start using the api calls I needed.     It would be worth it to see if 1.16 behaves the same way.     That flow is here:   https://github.com/cldr-steven-matison/NiFi-Templates/blob/main/NiFi_Template_XML_to_Flow_Definition_JSON.json     ... View more

@Fredb  This is a very difficult one to solve.      Does anyone know what would cause the execution of the sample_Import_Load.bat to run correctly from the windows command prompt, but fail when executed via the ExecuteStreamCommand processor with these errors?   This is most likely caused by permission issues.    Nifi requires specific permissions against files and scripts it touches or executes from within processors.  As such, the error is saying the processor does not know where any of the resources exist to run that .bat file.       I do not have any experience with nifi on windows, other than to avoid it, but the solution is likely the same as other operating systems.   Make sure the nifi user has full ownership of the file(s).   Additionally, it is sometimes possible to find deeper errors looking at the nifi-app.log file while testing and/or setting the log level of the processor to be more aggressive. ... View more

@rupeshh In order for nifi to be able to see the mounted directory or files within it, it needs to be properly owned to the same user that is running nifi.  For example:   chown nifi:nifi /some/path     Then nifi will be able to see the directories and files. ... View more