Dear Folks, i am having very strange issue with Flume NG, I could not get solution at anywhere, so i m reaching here. I have configured flume to bring my webserver log to HDFS. The problem here is, I am not seeing any error in Flume agent Log and Flume Collector Log. So i got strucked where to start diagnose the issue. Idea here is, i have webserver and I need to send those server logs to HDFS. i installed Flume NG agent and configured Sppol Source, File channel and Avro Sink. I installed Flume NG collector using Cloudera Manager and It has Avro Source, File channel and HDFS Sink. Flume NG version is 1.4.0+23-1.cdh (both web server and Collector having same version) NOTE: Webserver and Collector (Flume NG collector) are in public network , Means, Those two servers can be reachable over internet using one-one NATing. I did not see any issue in connectivity, I am able to see ESTABLISHED session when i do telnet from WEB server to COLLECTOR over port 9999. Flume NG agent config at WEB SERVER: ================================== # list sources, sinks and channels in the agent T2.sources = spool-source T2.sinks = avro-avsink T2.channels = file-channel # Source Property T2.sources.spool-source.type = spooldir T2.sources.spool-source.spoolDir = /logs-archive/test T2.sources.spool-source.channels = file-channel T2.sources.spool-source.fileHeader = true T2.sources.spool-source.trackerDir = /logs-archive/flume/DATA-03 # Sink Property T2.sinks.avro_avsink.type = avro T2.sinks.avro_avsink.channel = file-channel T2.sinks.avro_avsink.hostname = 208.XX.XX.XX # PUBLIC IP ADDRESS OF COLLECTOR T2.sinks.avro_avsink.port = 9999 # Channel Property T2.channels.file-channel.type = memory T2.channels.file-channel.checkpointDir = /logs-archive/flume/DATA-01 T2.channels.file-channel.dataDirs = /logs-archive/flume/DATA-02 FLUME NG COLLECTOR config : ========================= # agent name, in this case 'tier1'. tier1.sources = source1 tier1.channels = channel1 tier1.sinks = sink1 # For each source, channel, and sink, set # standard properties. tier1.sources.source1.type = avro tier1.sources.source1.bind = 10.2.4.10 # PRIVATE IP ADDRESS OF COLLECTOR, BUT IT HAS BEEN NAT WITH PUBLIC IP 208.XX.XX.XX tier1.sources.source1.port = 9999 tier1.sources.source1.channels = channel1 tier1.sources.source1.threads = 12 tier1.channels.channel1.type = file tier1.channels.channel1.checkpointDir = /mnt/sdd1 tier1.channels.channel1.dataDir = /mnt/sdd1 tier1.sinks.sink1.type = hdfs tier1.sinks.sink1.filePrefix = %{host} tier1.sinks.sink1.channel = channel1 tier1.sinks.sink1.hdfs.path = hdfs://xyz.com/user/test tier1.sinks.sink1.threadsPoolSize = 20 Any Advice or Suggestion are truly appreciated !!! Best Regards, Bommuraj
... View more
