New Contributor
Posts: 3
Registered: ‎11-07-2016

How to use Taildir source

Hi, Guys

I'm Trying to use taildir source for streaming logs.i have 500 records in log and new line will appened to this log, while running agent using taildir it reading 500 records perfectly.

while new records return to log it nor reading any data till log saved after log saved it reading from first. but i want to read only new records which recently written.

As I read In user guide taildir will set a file position after reading records from log, if any records written it will read those neew records using this file postion but it seems not working fine.


Please Help.


Myconfig file:

agent.sources = localsource
agent.channels = memoryChannel
agent.sinks = avro_Sink

# For each one of the sources, the type is defined
agent.sources.localsource.type = TAILDIR
agent.sources.localsource.positionFile = /home/admin1/teja/flume/taildir_position.json
agent.sources.localsource.filegroups = f1
agent.sources.localsource.filegroups.f1 = /home/admin1/teja/Flumedata/Behaviourlog.log
agent.sources.localsource.batchSize = 20
agent.sources.localsource.fileHeader = true

# The channel can be defined as follows.
agent.sources.localsource.channels = memoryChannel

#Specify the channel the sink should use = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 1000

# Each sink's type must be defined
agent.sinks.avro_Sink.type = avro
agent.sinks.avro_Sink.port= 8021
agent.sinks.avro_Sink.avro.batchSize = 20
agent.sinks.avro_Sink.avro.rollCount = 0
agent.sinks.avro_Sink.avro.rollSize = 143060835
agent.sinks.avro_Sink.avro.rollInterval = 0

Posts: 739
Registered: ‎05-16-2016

Re: How to use Taildir source

What type of file you are using ? 

read the flume api it seems its a preview feature and it does not work on windows

New Contributor
Posts: 3
Registered: ‎11-07-2016

Re: How to use Taildir source

Hi, Guna

Thanks a lot for your reply. I'm using it in ubuntu system so its working fine. and im reading data from text file.
Posts: 739
Registered: ‎05-16-2016

teRe: How to use Taildir source

Your welcome mate.

Could you explain me more about this statement "while new records return to log it nor reading any data till log saved after log saved it reading from first "  - I am sorry I couldnt understand . 

New Contributor
Posts: 3
Registered: ‎06-11-2018

Re: How to use Taildir source

I working on Flume to append the data from a local directory to HDFS using Flume Source TAILDIR.

My use case is to do Delta Load If the new line comes in the source file in local dir so that will append in hdfs.

This is my Flume Conf file :

#configure the agentagent.sources=r1

agent.sources.r1.positionFile = /home/swechchha/Documents/taildir_position.json
agent.sources.r1.filegroups.f1=/home/swechchha/Documents/spooldir/agent.sources.r1.batchSize = 20agent.sources.r1.writePosInterval=2000agent.sources.r1.maxBackoffSleep=5000agent.sources.r1.fileHeader = trueagent.sources.r1.channels=k1

agent.sinks.c1.hdfs.path=hdfs://localhost:8020/flume_sinkagent.sinks.c1.hdfs.batchSize = 1000agent.sinks.c1.hdfs.rollSize = 268435456agent.sinks.c1.hdfs.writeFormat=Text

while running flume command : flume-ng agent -n agent -c conf -f /home/swechchha/Documents/flumereal.conf