10-31-2017 10:41 AM - last edited on 10-31-2017 11:45 AM by cjervis
I want to stream data out of Hadoop into an other Big Data system. Is it possible using HDFS Java API as the Host Big data system supports Java.
Below are my Requirement:
1. I am NOT supposed to store the data which got streamed out of Hadoop. I got to hold them in Memory in the other system to which it was streamed to for some processing.
2. Should support large set of files/Data.
3. Minimal IO Cost.
4. Only Hadoop HDFS files will be subjected as Streaming data.
5. This should be accomplished in Java or in Python as the Host big data system supports Java and Python
What will be best options to carry out this.