I have read several other threads on Flume .tmp files but our configuration is different. Here is a snapshot of a few files - you'll see the .tmp file and final file are there - but in different sizes. I am unclear why the .tmp files remain and why they are different in size from the final file. Sometimes tmp file is larger, sometimes smaller
| -rw-r--r-- | hdfs | hdfs | 79.86 MB | 3 | 128 MB | FlumeData.1486544400005 |
| -rw-r--r-- | hdfs | hdfs | 81.45 MB | 3 | 128 MB | FlumeData.1486544400005.tmp |
| -rw-r--r-- | hdfs | hdfs | 81.38 MB | 3 | 128 MB | FlumeData.1486544400006 |
| -rw-r--r-- | hdfs | hdfs | 80.73 MB | 3 | 128 MB | FlumeData.1486544400006.tmp |
Configuration snippet - we tried a few different combinations and hit on this as a way to avoid time outs and network constraints - i see some of my notes didn't seem to remain consistent with the actual set values...
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://[name node serverr]:8020/data/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000 # Set rollsize to 126 MB to be slightly less than the block size, 132120576
TwitterAgent.sinks.HDFS.hdfs.rollSize = 132120576 # Set rollcount to 0 so that it does not roll on number of events, but size of sink file
TwitterAgent.sinks.HDFS.hdfs.rollCount = 20000 # Added rollInterval to 06 minutes (3600 seconds) to cap out the time interval the data is in memory
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 3600
TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true
TwitterAgent.sinks.HDFS.hdfs.appendTimeout = 10000
TwitterAgent.sinks.HDFS.hdfs.callTimeout = 60000
TwitterAgent.sinks.HDFS.hdfs.threadsPoolSize = 100
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000 # Increased transactionCapacity to 1000 from 100 to see if this solves the memory problem
TwitterAgent.channels.MemChannel.transactionCapacity = 10000
,
I have read several other threads on Flume .tmp files but our configuration is different. Here is a snapshot of a few files - you'll see the .tmp file and final file are there - but in different sizes. I am unclear why the .tmp files remain and why they are different in size from the final file. Sometimes tmp file is larger, sometimes smaller
| -rw-r--r-- | hdfs | hdfs | 79.86 MB | 3 | 128 MB | FlumeData.1486544400005 |
| -rw-r--r-- | hdfs | hdfs | 81.45 MB | 3 | 128 MB | FlumeData.1486544400005.tmp |
| -rw-r--r-- | hdfs | hdfs | 81.38 MB | 3 | 128 MB | FlumeData.1486544400006 |
| -rw-r--r-- | hdfs | hdfs | 80.73 MB | 3 | 128 MB | FlumeData.1486544400006.tmp |
Configuration snippet - we tried a few different combinations and hit on this as a way to avoid time outs and network constraints - i see some of my notes didn't seem to remain consistent with the actual set values...
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://[name node serverr]:8020/data/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
# Set rollsize to 126 MB to be slightly less than the block size, 132120576
TwitterAgent.sinks.HDFS.hdfs.rollSize = 132120576
# Set rollcount to 0 so that it does not roll on number of events, but size of sink file
TwitterAgent.sinks.HDFS.hdfs.rollCount = 20000
# Added rollInterval to 06 minutes (3600 seconds) to cap out the time interval the data is in memory
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 3600
TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true
TwitterAgent.sinks.HDFS.hdfs.appendTimeout = 10000
TwitterAgent.sinks.HDFS.hdfs.callTimeout = 60000
TwitterAgent.sinks.HDFS.hdfs.threadsPoolSize = 100
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
# Increased transactionCapacity to 1000 from 100 to see if this solves the memory problem
TwitterAgent.channels.MemChannel.transactionCapacity = 10000
