I am trying to implement a data migration strategy to migrate data from Teradata to Redshift using NiFi.
Here is the configuration I am using for NiFi cluster:
1) Encryption before PutS3Object :
Here we are using the EncyptContent processor (encrypt strategy using Nifi Legacy KDF, MD5_128AES algorithm) and decrypted after PutS3Object (using the same details used as earlier
except 'Decrypt' as the strategy. It fails at Redshift when I use the copy command to insert data from S3 to redshift as it is trying to read the encrypted file from S3. It throws me an error as 'Invalid operation'.
When I run the same on AWS Redshift, it does execute but no records are inserted into the target table.
2) Encryption just after reading data from source (in my case Teradata)
I am using the ExecuteSQL processor which reads data from the source and then converts source tables into CSV format using the ConvertRecord processor before we insert those files into an S3 bucket.
If I am using encrypt content processor after ExecuteSQL, it throws me an error when the ConvertRecord processor tries to read those files and gives an error as 'The incoming file is not a data file'
My intent is if I can encrypt the data end to end. I tried looking for different options but so far it isn't giving me the desired result. Maybe if I am missing any processor or ignoring any properties that can help me achieve encryption, not sure. Anyone who can guide me to the solution or can suggest a better pipeline to implement this type of data migration will be helpful