28785
DISCUSSIONS
102012
MEMBERS
3160
ARTICLES
I am seeing something odd with a simple flume agent. The agent launches without error and creates the first file to begin collecting the events.
but then the size of the file in HDFS never changes even though I can see the source file growing.
The second I stop the agent with [Ctrl + C], the file size in HDFS reflects the file growth.
In earlier testing, I could see the file in HDFS grow in size or the idleTimeout parameter would create a new file in 10 minutes with no activity.
There are no errors and the events are being collected but they do not appear to be written to HDFS. Additionally, the idleTimeout is being ignored.
Any thiughts on what might be hapening ?
See any errors in this conf file ?
Thanks for any help.
CentOS - 6.5
CDH 5
agent-3.channels = ch-3 agent-3.sources = src-3 agent-3.sinks = sink-3 agent-3.channels.ch-3.type = memory # must be > sink batch size agent-3.channels.ch-3.capacity = 1000 # must match sink batchSize agent-3.channels.ch-3.transactionCapacity = 200 agent-3.sources.src-3.type = exec agent-3.sources.src-3.command = tail -F /mnt/windows/ebiz/devinerb_tohost.txt agent-3.sources.src-3.channels = ch-3 agent-3.sources.src-3.inputCharset = ISO-8859-1 agent-3.sources.src-3.outputCharset = UTF-8 agent-3.sources.src-3.decodeErrorPolicy = FAIL agent-3.sinks.sink-3.channel = ch-3 agent-3.sinks.sink-3.type = hdfs agent-3.sinks.sink-3.hdfs.writeFormat = Text agent-3.sinks.sink-3.hdfs.fileType = DataStream agent-3.sinks.sink-3.hdfs.fileSuffix = .txt agent-3.sinks.sink-3.hdfs.rollCount = 0 agent-3.sinks.sink-3.hdfs.rollInterval = 0 agent-3.sinks.sink-3.hdfs.idleTimeout = 600 # create new file at 1 GB (not sure this is necessary) agent-3.sinks.sink-3.hdfs.rollSize = 10000000 # must match source transactionCapacity agent-3.sinks.sink-3.hdfs.batchSize = 200 agent-3.sinks.sink-3.hdfs.path = /user/choa/dev/philips_flume_exec