NiFi: How to detect updates to S3 files and insert the updated files into Aurora PostgreSQL

[#1]

Purpose

Detect updates to S3 files and insert the updated files into Aurora PostgreSQL with NiFi

Data flow

The finished dataflow

Process

1.) Download the dataflow file

(JSON file Import_S3_To_Aurora_PostgreSQL.json)

2.) Create a new processor group.

When creating this processor group, choose the following JSON file to upload.

Step 1: Choose processor group
Step 2 :
Step 3 : Finish upload.

3.) Install JDBC Driver

wget https://jdbc.postgresql.org/download/postgresql-42.7.3.jar
mkdir /tmp/nifi
mv postgresql-42.7.3.jar /tmp/nifi/

4.) Set parameters in NiFi

• Set ListS3 parameters

 

• S3 Access Key set

 

 

 

The values input was protected as access key/values are sensitive values.
Only "Sensitive value set" be shown.

 

4.2) Start the controllver service AWSCredentialsProviderControllerService (for saving AWS sensitive values)

 

 

4.3) Start the CSVReader Controller service

 

 

4.4) Start the JDBC Connection pool（DBCPConnectionPool-postgreSQL）service

 

5. ) Set bucket value

5.1) Fix the Bucket,Prefix values

 

 

5.2) Fix the PostgreSQL table name for INSERT
PutDatabaseRecord setting:

 

6.) Start the processors

 

7.) Check the provenance: