Member since
07-30-2013
8
Posts
1
Kudos Received
2
Solutions
12-09-2013
11:27 AM
Dear Folks,We Fixed the issue with below configuaration. Thank you all for your assistance on this. FLUME agent Configuaration: ======================== # list sources, sinks and channels in the agent bench.sources = spool-source bench.sinks = avro-avsink bench.channels = file-channel # Source Property bench.sources.spool-source.type = spooldir bench.sources.spool-source.spoolDir = /logs-archive/test bench.sources.spool-source.channels = file-channel bench.sources.spool-source.batchSize = 2000 bench.sources.spool-source.fileHeader = true bench.sources.spool-source.fileHeaderKey = file bench.sources.spool-source.bufferMaxLineLength = 8000 bench.sources.spool-source.trackerDir = /logs-archive/flume/DATA-03 # Interceptor bench.sources.spool-source.interceptors = hostint bench.sources.spool-source.interceptors.hostint.type = org.apache.flume.interceptor.HostInterceptor$Builder bench.sources.spool-source.interceptors.hostint.preserveExisting = true bench.sources.spool-source.interceptors.hostint.useIP = false bench.sources.spool-source.interceptors = eventint bench.sources.spool-source.interceptors.eventint.type = AddEventHeader$Builder # Sink Property bench.sinks.avro-avsink.type = avro bench.sinks.avro-avsink.channel = file-channel bench.sinks.avro-avsink.hostname = flume-01.XYZ.com bench.sinks.avro-avsink.port = 9999 bench.sinks.avro_avsink.batch-size = 2000 # Channel Property bench.channels.file-channel.type = file bench.channels.file-channel.checkpointDir = /logs-archive/flume/DATA-01 bench.channels.file-channel.dataDirs = /logs-archive/flume/DATA-02 bench.channels.file-channel.transactionCapacity = 4000 bench.channels.file-channel.maxFileSize = 2147483648 bench.channels.file-channel.minimumRequiredSpace = 400000000 bench.channels.file-channel.capacity = 1000000 Flume Server Configuaration : ======================= # agent name, in this case 'tier1'. tier1.sources = source1 tier1.channels = channel1 tier1.sinks = sink1 # Source Property tier1.sources.source1.type = avro tier1.sources.source1.bind = flume-01.XYZ.com tier1.sources.source1.port = 9999 tier1.sources.source1.channels = channel1 tier1.sources.source1.threads = 12 # Channel Property tier1.channels.channel1.type = file tier1.channels.channel1.checkpointDir = /mnt/sdd1/checkpointDir tier1.channels.channel1.dataDirs = /mnt/sdd1/dataDir tier1.channels.channel1.minimumRequiredSpace = 83558400 tier1.channels.channel1.transactionCapacity = 1000000 tier1.channels.channel1.maxFileSize = 2147483648 tier1.channels.channel1.capacity = 9000000 # Interceptor Property #tier1.sources.source1.interceptors = hostint #tier1.sources.source1.interceptors.hostint.type = org.apache.flume.interceptor.HostInterceptor$Builder #tier1.sources.source1.interceptors.hostint.preserveExisting = true #tier1.sources.source1.interceptors.hostint.useIP = false # Sink Property tier1.sinks.sink1.type = hdfs tier1.sinks.sink1.hdfs.path = hdfs://example.intranet.com:8020/flume-test/%{eventName}/%Y-%m-%d/%H tier1.sinks.sink1.channel = channel1 tier1.sinks.sink1.hdfs.useLocalTimeStamp = true tier1.sinks.sink1.hdfs.filePrefix = %{host} tier1.sinks.sink1.hdfs.rollSize = 1048576 tier1.sinks.sink1.hdfs.rollInterval = 0 tier1.sinks.sink1.hdfs.rollCount = 0 tier1.sinks.sink1.hdfs.rollTimerPoolSize = 10 tier1.sinks.sink1.hdfs.batchSize = 20000 tier1.sinks.sink1.hdfs.maxOpenFiles = 500000 tier1.sinks.sink1.hdfs.file.Type = SequenceFile tier1.sinks.sink1.hdfs.writeFormat = Writable tier1.sinks.sink1.hdfs.threadsPoolSize = 20 Best Regards, Bommuraj
... View more