question NI-FI processors to get a file from HDFS and invoke an API via https in Archives of Support Questions (Read Only)

NI-FI processors to get a file from HDFS and invoke an API via https

Srikaran — Tue, 05 Dec 2017 03:17:20 GMT

Hello. I have a HDFS file for which data needs to be posted to an outside URL (https), I have the user name and password for the URL; I can post a sample JSON via postman from my browser by using the user name and password. Now I have to use Ni-FI for this flow. Please let me know what are the exact nifi processors should I use to get the data from HDFS and post it into the URL via another ni-fi processor. Also kindly let me know what format the HDFS data should be in for these kind of use-cases. Thanks Srikaran

Re: NI-FI processors to get a file from HDFS and invoke an API via https

anarasimham — Wed, 06 Dec 2017 23:20:44 GMT

You will need to use the GetHDFS processor to retrieve the file and then the InvokeHTTP processor to send the data to an HTTP endpoint. Data format shouldn't matter - a binary sequence is being transmitted so unless you need to parse the data before transmission it can be anything.

If you are dealing with a large file, you may want to split it as you could run into memory limitations. You will have to split it before transmission into manageable chunks and join it afterwards.

Re: NI-FI processors to get a file from HDFS and invoke an API via https

TimothySpann — Thu, 07 Dec 2017 00:57:23 GMT

Some Apache NiFi examples that may be helpful

https://community.hortonworks.com/articles/47854/accessing-facebook-page-data-from-apache-nifi.html

https://community.hortonworks.com/articles/86570/hosting-and-ingesting-data-from-web-pages-desktop.html

https://community.hortonworks.com/content/kbentry/65239/mp3-jukebox-with-nifi-1x.html

https://community.hortonworks.com/articles/65154/nifi-1x-for-automatic-music-playing-pipelines.html

https://community.hortonworks.com/articles/73355/adding-a-custom-processor-to-nifi-linkprocessor.html

Re: NI-FI processors to get a file from HDFS and invoke an API via https

Srikaran — Thu, 07 Dec 2017 02:24:47 GMT

@anarasimham Looks like GetHDFS will replace HDFS file. I am planning to use fetchHDFS and then invoke http processor. For now I am converting avro file to JSON on Hadoop end and fetching the json and posting it. I will directly test avro & other formats and will update. Thanks!

Re: NI-FI processors to get a file from HDFS and invoke an API via https

Srikaran — Thu, 07 Dec 2017 02:26:21 GMT

@Timothy Spann

Thanks a lot. These are very helpful, Let me test the flow and will update accordingly. Thanks