question Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. in Archives of Support Questions (Read Only) https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133962#M47762 <P>What is the largest load (MBs or GBs) you have run your use case on?</P> Sat, 17 Dec 2016 02:09:29 GMT gkeys 2016-12-17T02:09:29Z Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133955#M47755 Fri, 02 Dec 2016 06:15:26 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133955#M47755 raviteja_gogata 2016-12-02T06:15:26Z Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133956#M47756 <P>@<A href="https://community.hortonworks.com/users/11346/vijaykumar243.html">Vijay Kumar J</A> any idea ? thanks in advance</P> Sun, 04 Dec 2016 08:30:11 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133956#M47756 raviteja_gogata 2016-12-04T08:30:11Z Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133957#M47757 <P>@Greg Keys any idea ? thanks in advance</P> Fri, 16 Dec 2016 00:48:56 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133957#M47757 raviteja_gogata 2016-12-16T00:48:56Z Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133958#M47758 <P>I suggest looking at the merge and saveAsTextFile functions as per bottom post here <A href="http://stackoverflow.com/questions/31666361/process-spark-streaming-rdd-and-store-to-single-hdfs-file" target="_blank">http://stackoverflow.com/questions/31666361/process-spark-streaming-rdd-and-store-to-single-hdfs-file</A></P> Fri, 16 Dec 2016 05:23:32 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133958#M47758 gkeys 2016-12-16T05:23:32Z Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133959#M47759 <P>Hi Greg Keys, thanks for the reply i was using the similar approach, but wondering whether this approach works if spark streaming processing the data in giga bytes ?</P> Fri, 16 Dec 2016 05:58:38 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133959#M47759 raviteja_gogata 2016-12-16T05:58:38Z Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133960#M47760 <P>That is really an issue of scaling (how many nodes and memory per node you have) and multitenancy (which other jobs will run at the same time, particularly spark or other memory-intensive jobs). The more nodes and the less memory contention, the more data you can process in spark.</P> Fri, 16 Dec 2016 21:47:53 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133960#M47760 gkeys 2016-12-16T21:47:53Z Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133961#M47761 <P>i am working on 12 node cluster with 4 having 126 gigs, 8 having 252 gigs memory. </P> Sat, 17 Dec 2016 02:07:01 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133961#M47761 raviteja_gogata 2016-12-17T02:07:01Z Re: Is it possible to write the spark streaming output to single file in HDFS ? where spark streaming get's the logs from kafka topics. https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133962#M47762 <P>What is the largest load (MBs or GBs) you have run your use case on?</P> Sat, 17 Dec 2016 02:09:29 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Is-it-possible-to-write-the-spark-streaming-output-to-single/m-p/133962#M47762 gkeys 2016-12-17T02:09:29Z