question Re: Flume is leaving .tmp files,Flume leaving .tmp files in place in Archives of Support Questions (Read Only) https://community.cloudera.com/t5/Archives-of-Support-Questions/Flume-is-leaving-tmp-files-Flume-leaving-tmp-files-in-place/m-p/167161#M53846 <P>Thank you - turns out for us, I had mistakenly started up a second flume instance which somehow was colliding with the first. User error. </P> Thu, 16 Feb 2017 00:00:37 GMT cthomas1 2017-02-16T00:00:37Z Flume is leaving .tmp files,Flume leaving .tmp files in place https://community.cloudera.com/t5/Archives-of-Support-Questions/Flume-is-leaving-tmp-files-Flume-leaving-tmp-files-in-place/m-p/167159#M53844 <P>I have read several other threads on Flume .tmp files but our configuration is different. Here is a snapshot of a few files - you'll see the .tmp file and final file are there - but in different sizes. I am unclear why the .tmp files remain and why they are different in size from the final file. Sometimes tmp file is larger, sometimes smaller</P><TABLE><TBODY><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>79.86 MB</TD><TD>3</TD><TD>128 MB</TD><TD>FlumeData.1486544400005</TD></TR><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>81.45 MB</TD><TD>3</TD><TD>128 MB</TD><TD>FlumeData.1486544400005.tmp</TD></TR><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>81.38 MB</TD><TD>3</TD><TD>128 MB</TD><TD>FlumeData.1486544400006</TD></TR><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>80.73 MB</TD><TD>3</TD><TD>128 MB</TD><TD><P>FlumeData.1486544400006.tmp</P></TD></TR></TBODY></TABLE><P>Configuration snippet - we tried a few different combinations and hit on this as a way to avoid time outs and network constraints - i see some of my notes didn't seem to remain consistent with the actual set values...</P><P>TwitterAgent.sinks.HDFS.channel = MemChannel </P><P>TwitterAgent.sinks.HDFS.type = hdfs </P><P>TwitterAgent.sinks.HDFS.hdfs.path = hdfs://[name node serverr]:8020/data/tweets/%Y/%m/%d/%H/ </P><P>TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream </P><P>TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text </P><P>TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000 # Set rollsize to 126 MB to be slightly less than the block size, 132120576 </P><P>TwitterAgent.sinks.HDFS.hdfs.rollSize = 132120576 # Set rollcount to 0 so that it does not roll on number of events, but size of sink file </P><P>TwitterAgent.sinks.HDFS.hdfs.rollCount = 20000 # Added rollInterval to 06 minutes (3600 seconds) to cap out the time interval the data is in memory </P><P>TwitterAgent.sinks.HDFS.hdfs.rollInterval = 3600 </P><P>TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true </P><P>TwitterAgent.sinks.HDFS.hdfs.appendTimeout = 10000 </P><P>TwitterAgent.sinks.HDFS.hdfs.callTimeout = 60000 </P><P>TwitterAgent.sinks.HDFS.hdfs.threadsPoolSize = 100 </P><P>TwitterAgent.channels.MemChannel.type = memory </P><P>TwitterAgent.channels.MemChannel.capacity = 10000 # Increased transactionCapacity to 1000 from 100 to see if this solves the memory problem </P><P>TwitterAgent.channels.MemChannel.transactionCapacity = 10000</P>,<P>I have read several other threads on Flume .tmp files but our configuration is different. Here is a snapshot of a few files - you'll see the .tmp file and final file are there - but in different sizes. I am unclear why the .tmp files remain and why they are different in size from the final file. Sometimes tmp file is larger, sometimes smaller</P><TABLE><TBODY><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>79.86 MB</TD><TD>3</TD><TD>128 MB</TD><TD>FlumeData.1486544400005</TD></TR><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>81.45 MB</TD><TD>3</TD><TD>128 MB</TD><TD>FlumeData.1486544400005.tmp</TD></TR><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>81.38 MB</TD><TD>3</TD><TD>128 MB</TD><TD>FlumeData.1486544400006</TD></TR><TR><TD>-rw-r--r--</TD><TD>hdfs</TD><TD>hdfs</TD><TD>80.73 MB</TD><TD>3</TD><TD>128 MB</TD><TD><P>FlumeData.1486544400006.tmp</P></TD></TR></TBODY></TABLE><P>Configuration snippet - we tried a few different combinations and hit on this as a way to avoid time outs and network constraints - i see some of my notes didn't seem to remain consistent with the actual set values...</P><P>TwitterAgent.sinks.HDFS.channel = MemChannel </P><P>TwitterAgent.sinks.HDFS.type = hdfs </P><P>TwitterAgent.sinks.HDFS.hdfs.path = hdfs://[name node serverr]:8020/data/tweets/%Y/%m/%d/%H/ </P><P>TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream </P><P>TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text </P><P>TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000 # Set rollsize to 126 MB to be slightly less than the block size, 132120576 </P><P>TwitterAgent.sinks.HDFS.hdfs.rollSize = 132120576 # Set rollcount to 0 so that it does not roll on number of events, but size of sink file </P><P>TwitterAgent.sinks.HDFS.hdfs.rollCount = 20000 # Added rollInterval to 06 minutes (3600 seconds) to cap out the time interval the data is in memory </P><P>TwitterAgent.sinks.HDFS.hdfs.rollInterval = 3600 </P><P>TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true </P><P>TwitterAgent.sinks.HDFS.hdfs.appendTimeout = 10000 </P><P>TwitterAgent.sinks.HDFS.hdfs.callTimeout = 60000 </P><P>TwitterAgent.sinks.HDFS.hdfs.threadsPoolSize = 100 </P><P>TwitterAgent.channels.MemChannel.type = memory </P><P>TwitterAgent.channels.MemChannel.capacity = 10000 # Increased transactionCapacity to 1000 from 100 to see if this solves the memory problem </P><P>TwitterAgent.channels.MemChannel.transactionCapacity = 10000</P> Thu, 09 Feb 2017 07:31:41 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Flume-is-leaving-tmp-files-Flume-leaving-tmp-files-in-place/m-p/167159#M53844 cthomas1 2017-02-09T07:31:41Z Re: Flume is leaving .tmp files,Flume leaving .tmp files in place https://community.cloudera.com/t5/Archives-of-Support-Questions/Flume-is-leaving-tmp-files-Flume-leaving-tmp-files-in-place/m-p/167160#M53845 <P><A rel="user" href="https://community.cloudera.com/users/15924/cthomas1.html" nodeid="15924">@Cord thomas </A></P><P>Turn on debug logging and check the log file first <A rel="user" href="https://community.cloudera.com/users/15924/cthomas1.html" nodeid="15924"></A> </P> Wed, 15 Feb 2017 23:52:01 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Flume-is-leaving-tmp-files-Flume-leaving-tmp-files-in-place/m-p/167160#M53845 bluesmix 2017-02-15T23:52:01Z Re: Flume is leaving .tmp files,Flume leaving .tmp files in place https://community.cloudera.com/t5/Archives-of-Support-Questions/Flume-is-leaving-tmp-files-Flume-leaving-tmp-files-in-place/m-p/167161#M53846 <P>Thank you - turns out for us, I had mistakenly started up a second flume instance which somehow was colliding with the first. User error. </P> Thu, 16 Feb 2017 00:00:37 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Flume-is-leaving-tmp-files-Flume-leaving-tmp-files-in-place/m-p/167161#M53846 cthomas1 2017-02-16T00:00:37Z