New Contributor
Posts: 1
Registered: ‎10-31-2017

Streaming Data Out of Hadoop to other Big Data system

[ Edited ]

I want to stream data out of Hadoop into an other Big Data system. Is it possible using HDFS Java API as the Host Big data system supports Java.


Below are my Requirement:


1.  I am NOT supposed to store the data which got streamed out of Hadoop. I got to hold them in Memory in the other system to which it was streamed to for some processing.

2.  Should support large set of files/Data.

3. Minimal IO Cost.

4. Only Hadoop HDFS files will be subjected as Streaming data.

5. This should be accomplished in Java or in Python as the Host big data system supports Java and Python 


What will be best options to carry out this.