NI-FI processors to get a file from HDFS and invoke an API via https

[#1]

Hello. I have a HDFS file for which data needs to be posted to an outside URL (https), I have the user name and password for the URL; I can post a sample JSON via postman from my browser by using the user name and password. Now I have to use Ni-FI for this flow. Please let me know what are the exact nifi processors should I use to get the data from HDFS and post it into the URL via another ni-fi processor. Also kindly let me know what format the HDFS data should be in for these kind of use-cases. Thanks Srikaran

[#2 · Accepted solution]

You will need to use the GetHDFS processor to retrieve the file and then the InvokeHTTP processor to send the data to an HTTP endpoint. Data format shouldn't matter - a binary sequence is being transmitted so unless you need to parse the data before transmission it can be anything.

If you are dealing with a large file, you may want to split it as you could run into memory limitations. You will have to split it before transmission into manageable chunks and join it afterwards.

[#3]

@anarasimham Looks like GetHDFS will replace HDFS file. I am planning to use fetchHDFS and then invoke http processor. For now I am converting avro file to JSON on Hadoop end and fetching the json and posting it. I will directly test avro & other formats and will update. Thanks!

[#4]

Some Apache NiFi examples that may be helpful

https://community.hortonworks.com/articles/47854/accessing-facebook-page-data-from-apache-nifi.html

https://community.hortonworks.com/articles/86570/hosting-and-ingesting-data-from-web-pages-desktop.h...

https://community.hortonworks.com/content/kbentry/65239/mp3-jukebox-with-nifi-1x.html

https://community.hortonworks.com/articles/65154/nifi-1x-for-automatic-music-playing-pipelines.html

https://community.hortonworks.com/articles/73355/adding-a-custom-processor-to-nifi-linkprocessor.htm...

[#5]

@Timothy Spann

Thanks a lot. These are very helpful, Let me test the flow and will update accordingly. Thanks