Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Expert Contributor

Purpose

Detect updates to S3 files and insert the updated files into Aurora PostgreSQL with NiFi

Data flow

The finished dataflow

b9fd6e9c-6cd6-b2fd-32c6-77e68c2a00ec

Process

1.) Download the dataflow file

(JSON file Import_S3_To_Aurora_PostgreSQL.json)

2.) Create a new processor group.

When creating this processor group, choose the following JSON file to upload.
Step 1: Choose processor group
Step 2 :
Step 3 : Finish upload.

3.) Install JDBC Driver

wget https://jdbc.postgresql.org/download/postgresql-42.7.3.jar
mkdir /tmp/nifi
mv postgresql-42.7.3.jar /tmp/nifi/

4.) Set parameters in NiFi

  • Set ListS3 parameters
 
  • S3 Access Key set
 
 
 

The values input was protected as access key/values are sensitive values.
Only "Sensitive value set" be shown.

 

4.2) Start the controllver service AWSCredentialsProviderControllerService (for saving AWS sensitive values)

 
 

4.3) Start the CSVReader Controller service

 

 

 

4.4) Start the JDBC Connection pool(DBCPConnectionPool-postgreSQL)service

 

5. ) Set bucket value

5.1) Fix the Bucket,Prefix values

 

 

 

5.2) Fix the PostgreSQL table name for INSERT
PutDatabaseRecord setting:

 

6.) Start the processors

 

7.) Check the provenance:

 
 

 

233 Views