Created on 07-23-201803:15 PM - edited 08-17-201906:45 AM
Nifi Flow for
writing to S3, WASB and Google Storage.
Run the flow, watch as the twitter messages are
captured and then aggregated before putting them in storage.
Now you can go to your Azure Portal and look in the
container and you should see aggregated messages organized by year/month/day.
Open Google Cloud Platform and go to your Storage service. Google Storage
will now contain. the aggregated messages organized by year/month/day.
S3 bucket in your AWS account will now have aggregated
Twitter messages organized by year/month/day.
Now let’s see what’s happening here. I will only focus on the three main important processors as the others make up the simple flow. The entire flow template is available as an xml file and you can download: nificloudstorage.xml
Create a Storage Account
Get the Storage Account name and Key as shown in this screenshot. This is needed in the PutAzureObject Processor.
From AWS dashboard, go to Users, pick your user, click on Security Credentials. If you have not saved the Secret Access key then use Create Access key button to generate it again. There is a limit of only 2 keys.
Setting up GCS credentials is slightly different. A Controller Service is made use of. Click on the arrow in GCPCredentialsControllerService. That will take you to the next screenshot.
Controller Services Click on the gears icon to take you to the properties.
Use the JSON file created from your GCS credentials. You can follow this article Creating GCS credentials to find out to get this JSON
Click on the lightning icon to enable this Controller Service