Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Contributor

Nifi Flow for writing to S3, WASB and Google Storage.

82403-nifis3flowpart1.png

82404-nifis3flowpart2.png

Run the flow, watch as the twitter messages are captured and then aggregated before putting them in storage.


Azure Storage:

Now you can go to your Azure Portal and look in the container and you should see aggregated messages organized by year/month/day.

82411-wasbstorageresult.png

Google Storage:

Open Google Cloud Platform and go to your Storage service. Google Storage will now contain. the aggregated messages organized by year/month/day.

82410-gcsstorageresult.png

AWS Storage:

S3 bucket in your AWS account will now have aggregated Twitter messages organized by year/month/day.

82409-s3storageresult.png

Now let’s see what’s happening here. I will only focus on the three main important processors as the others make up the simple flow. The entire flow template is available as an xml file and you can download: nificloudstorage.xml

PutAzureStorage Processor:

82415-putazurestorage.png

Azure:

Create a Storage Account

82416-createstorageacct.png

Get the Storage Account name and Key as shown in this screenshot. This is needed in the PutAzureObject Processor.

82417-storageaccount.png

PutS3Object Processor

82418-puts3object.png

From AWS dashboard, go to Users, pick your user, click on Security Credentials. If you have not saved the Secret Access key then use Create Access key button to generate it again. There is a limit of only 2 keys.

82419-s3accesskeysecret.png

PutGCSObject Processor

Setting up GCS credentials is slightly different. A Controller Service is made use of. Click on the arrow in GCPCredentialsControllerService. That will take you to the next screenshot.

82420-putgcssobject.png

Controller Services
Click on the gears icon to take you to the properties.

82421-gcscontrollerservice.png

Use the JSON file created from your GCS credentials. You can follow this article Creating GCS credentials to find out to get this JSON

82422-controllerserviceproperty.png

Click on the lightning icon to enable this Controller Service

82423-enablecontrollerservice.png


wasbstorageresult.png82414storageaccount.pngstorageaccount.png82412createstorageacct.pngs3storageresult.pnggcsstorageresult.pngcreatestorageacct.png
4,614 Views
Comments
avatar
New Contributor

I am using nifi-1.11.4 and was able to upload the xml.  When I try to use the xml in my canvas I get error 

org.apache.nifi.processors.kite.ConvertJSONToAvro is not known to this NiFi instance.  I would really like to try out this template. What is the solution.

avatar
Moderator

Hello @MeenaK ,

 

thank you for reaching out.

 

This community thread mentions, that "The ConvertJsonToAvro processor was removed from the default NiFi distribution bundle because of space limitations as of the Apache NiFi 1.10 release.".

 

this thread points to a repo of the Apache Nifi.

 

Hope it helps!

 

Kind regards:

Ferenc

Version history
Last update:
‎08-17-2019 06:45 AM
Updated by:
Contributors