Hi,   I have the solution. Please check my post in stackoverflow: http://stackoverflow.com/questions/37466361/how-to-combine-two-dstreams-using-pyspark-similar-to-zip-on-normal-rdd/37537555#37537555   Thanks, Obaid ... View more

Hi,   Sorry guys for the reply whitch is too late.   Anyways, I tried with different combinition(memory/disk channel etc.) and found flume is either failing of too slow to  load larger files (more that 1G). So, I conclude that flume is not good for lage files.   Instead, I am now using HDFS NFS gateways to dump file directly to HDFS using scp.  Belive me, correctly configured NFS GW and NFS mount point are really cool old boys.   Thanks, Obaid  ... View more