Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NI-FI processors to get a file from HDFS and invoke an API via https

avatar
Expert Contributor

Hello. I have a HDFS file for which data needs to be posted to an outside URL (https), I have the user name and password for the URL; I can post a sample JSON via postman from my browser by using the user name and password. Now I have to use Ni-FI for this flow. Please let me know what are the exact nifi processors should I use to get the data from HDFS and post it into the URL via another ni-fi processor. Also kindly let me know what format the HDFS data should be in for these kind of use-cases. Thanks Srikaran

1 ACCEPTED SOLUTION

avatar
Super Collaborator

You will need to use the GetHDFS processor to retrieve the file and then the InvokeHTTP processor to send the data to an HTTP endpoint. Data format shouldn't matter - a binary sequence is being transmitted so unless you need to parse the data before transmission it can be anything.

If you are dealing with a large file, you may want to split it as you could run into memory limitations. You will have to split it before transmission into manageable chunks and join it afterwards.

View solution in original post

4 REPLIES 4

avatar
Super Collaborator

You will need to use the GetHDFS processor to retrieve the file and then the InvokeHTTP processor to send the data to an HTTP endpoint. Data format shouldn't matter - a binary sequence is being transmitted so unless you need to parse the data before transmission it can be anything.

If you are dealing with a large file, you may want to split it as you could run into memory limitations. You will have to split it before transmission into manageable chunks and join it afterwards.

avatar
Expert Contributor

@anarasimham Looks like GetHDFS will replace HDFS file. I am planning to use fetchHDFS and then invoke http processor. For now I am converting avro file to JSON on Hadoop end and fetching the json and posting it. I will directly test avro & other formats and will update. Thanks!

avatar
Expert Contributor
@Timothy Spann

Thanks a lot. These are very helpful, Let me test the flow and will update accordingly. Thanks