Posts: 56
Registered: ‎02-09-2015

Spark Streaming Visualization

I am trying to visualize data processed by spark , but i can't find any tools to do , i am doing word count example using spark , then try to visualize these data (i.e: do word count for the coming stream using spark then visualize it , and keep doing so for the next streams ) 

my pipeline goes like this :


1- flume stream ( source is spool directory having text files )

2- flume sink is spark , so spark take the flume stream and do word count

3- spark then save the output data in HDFS 


so i want to visualize the spark output , i thought about 2 methods 


1- try to find visualizer to crawl the HDFS directory continously and visualize the content (can't find a visualizer to do so )

2- save the output in mysql database , then find any visualizer to visualize the mysql database (on long term it wont be efficient to use mysql to store large amount of data and do query on it to do the visualization)



so folks , anyone have any information or experience about this issue ? or tried to visualize spark output ?


Thanks in advance