Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Cloudera Employee

Nifi Flow for writing to S3, WASB and Google Storage.

82403-nifis3flowpart1.png

82404-nifis3flowpart2.png

Run the flow, watch as the twitter messages are captured and then aggregated before putting them in storage.


Azure Storage:

Now you can go to your Azure Portal and look in the container and you should see aggregated messages organized by year/month/day.

82411-wasbstorageresult.png

Google Storage:

Open Google Cloud Platform and go to your Storage service. Google Storage will now contain. the aggregated messages organized by year/month/day.

82410-gcsstorageresult.png

AWS Storage:

S3 bucket in your AWS account will now have aggregated Twitter messages organized by year/month/day.

82409-s3storageresult.png

Now let’s see what’s happening here. I will only focus on the three main important processors as the others make up the simple flow. The entire flow template is available as an xml file and you can download: nificloudstorage.xml

PutAzureStorage Processor:

82415-putazurestorage.png

Azure:

Create a Storage Account

82416-createstorageacct.png

Get the Storage Account name and Key as shown in this screenshot. This is needed in the PutAzureObject Processor.

82417-storageaccount.png

PutS3Object Processor

82418-puts3object.png

From AWS dashboard, go to Users, pick your user, click on Security Credentials. If you have not saved the Secret Access key then use Create Access key button to generate it again. There is a limit of only 2 keys.

82419-s3accesskeysecret.png

PutGCSObject Processor

Setting up GCS credentials is slightly different. A Controller Service is made use of. Click on the arrow in GCPCredentialsControllerService. That will take you to the next screenshot.

82420-putgcssobject.png

Controller Services
Click on the gears icon to take you to the properties.

82421-gcscontrollerservice.png

Use the JSON file created from your GCS credentials. You can follow this article Creating GCS credentials to find out to get this JSON

82422-controllerserviceproperty.png

Click on the lightning icon to enable this Controller Service

82423-enablecontrollerservice.png


wasbstorageresult.png82414storageaccount.pngstorageaccount.png82412createstorageacct.pngs3storageresult.pnggcsstorageresult.pngcreatestorageacct.png
1,524 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 06:45 AM
Updated by:
 
Contributors
Top Kudoed Authors