Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to use Taildir source

How to use Taildir source

New Contributor

Hi, Guys

I'm Trying to use taildir source for streaming logs.i have 500 records in log and new line will appened to this log, while running agent using taildir it reading 500 records perfectly.

while new records return to log it nor reading any data till log saved after log saved it reading from first. but i want to read only new records which recently written.

As I read In user guide taildir will set a file position after reading records from log, if any records written it will read those neew records using this file postion but it seems not working fine.


Please Help.


Myconfig file:

agent.sources = localsource
agent.channels = memoryChannel
agent.sinks = avro_Sink

# For each one of the sources, the type is defined
agent.sources.localsource.type = TAILDIR
agent.sources.localsource.positionFile = /home/admin1/teja/flume/taildir_position.json
agent.sources.localsource.filegroups = f1
agent.sources.localsource.filegroups.f1 = /home/admin1/teja/Flumedata/Behaviourlog.log
agent.sources.localsource.batchSize = 20
agent.sources.localsource.fileHeader = true

# The channel can be defined as follows.
agent.sources.localsource.channels = memoryChannel

#Specify the channel the sink should use = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 1000

# Each sink's type must be defined
agent.sinks.avro_Sink.type = avro
agent.sinks.avro_Sink.port= 8021
agent.sinks.avro_Sink.avro.batchSize = 20
agent.sinks.avro_Sink.avro.rollCount = 0
agent.sinks.avro_Sink.avro.rollSize = 143060835
agent.sinks.avro_Sink.avro.rollInterval = 0


Re: How to use Taildir source


What type of file you are using ? 

read the flume api it seems its a preview feature and it does not work on windows

Re: How to use Taildir source

New Contributor
Hi, Guna

Thanks a lot for your reply. I'm using it in ubuntu system so its working fine. and im reading data from text file.

teRe: How to use Taildir source


Your welcome mate.

Could you explain me more about this statement "while new records return to log it nor reading any data till log saved after log saved it reading from first "  - I am sorry I couldnt understand . 


Re: How to use Taildir source

New Contributor

I working on Flume to append the data from a local directory to HDFS using Flume Source TAILDIR.

My use case is to do Delta Load If the new line comes in the source file in local dir so that will append in hdfs.

This is my Flume Conf file :

#configure the agentagent.sources=r1

agent.sources.r1.positionFile = /home/swechchha/Documents/taildir_position.json
agent.sources.r1.filegroups.f1=/home/swechchha/Documents/spooldir/agent.sources.r1.batchSize = 20agent.sources.r1.writePosInterval=2000agent.sources.r1.maxBackoffSleep=5000agent.sources.r1.fileHeader = trueagent.sources.r1.channels=k1

agent.sinks.c1.hdfs.path=hdfs://localhost:8020/flume_sinkagent.sinks.c1.hdfs.batchSize = 1000agent.sinks.c1.hdfs.rollSize = 268435456agent.sinks.c1.hdfs.writeFormat=Text

while running flume command : flume-ng agent -n agent -c conf -f /home/swechchha/Documents/flumereal.conf

Re: How to use Taildir source

New Contributor

Where you able to solve that problem?  Seeing the same behavior.

Don't have an account?
Coming from Hortonworks? Activate your account here