New Contributor
Posts: 3
Registered: ‎05-10-2017

Flume Agent error : avro-source has been removed due to an error during configuration

Hi All,


I am trying to ingest logs from remote server(Azure VM) on to my quickstart VM and have done the following :


Agent 1(to be run on remote server) - Exec source, file channel and Avro sink

Agent 2 (to be run on quickstart vm) - Avro source, file channel, hdfs sink


I am getting the error "Source avro-source has been removed due to an error during configuration" while I initiate Agent 2.


Can anyone help in solving this. I am not able to figure out the issue in conf file.


Conf file on Quickstart vm:


# Name the components on this agent
Agent2.sources = avro-source
Agent2.channels = file-channel
Agent2.sinks = hdfs-sink

# Describe/configure Source
Agent2.sources.avro-source.type = avro
Agent2.sources.avro-source.hostname = quickstart.cloudera
Agent2.sources.avro-source.port = 9083

# Describe the sink
Agent2.sinks.hdfs-sink.type = hdfs
Agent2.sinks.hdfs-sink.hdfs.path = hdfs://quickstart.cloudera:8020/user/cloudera/flumetest/Controller_%Y-%m-%d
Agent2.sinks.hdfs-sink.hdfs.rollInterval = 120
Agent2.sinks.hdfs-sink.hdfs.rollSize = 10485760
Agent2.sinks.hdfs-sink.hdfs.rollCount = 10000
Agent2.sinks.hdfs-sink.hdfs.fileType = DataStream
Agent2.sinks.hdfs-sink.hdfs.filePrefix = log
Agent2.sinks.hdfs-sink.hdfs.fileSuffix = .txt
Agent2.sinks.hdfs-sink.hdfs.inUseSuffix = .tmp
Agent2.sinks.hdfs-sink.hdfs.useLocalTimeStamp = true

#Use a channel which buffers events in file
Agent2.channels.file-channel.type = file
Agent2.channels.file-channel.checkpointDir = /home/cloudera/Projects/flumetest/checkpoint/
Agent2.channels.file-channel.dataDirs = /home/cloudera/Projects/flumetest/data/

# Bind the source and sink to the channel
Agent2.sources.avro-source.channels = file-channel = file-channel


Conf File on Remote Server



Agent1.sources = logsource
Agent1.channels = filechannel
Agent1.sinks = avrosink


# Describe / Configure the source
Agent1.sources.logsource.type = exec
Agent1.sources.logsource.command = tail -F /home/dev/projects/logData/node-PlatformVM.log


#Describe the sink
Agent1.sinks.avro-sink.type = avro
Agent1.sinks.avro-sink.hostname = quickstart.cloudera
Agent1.sinks.avro-sink.port = 9083


#Use a channel which buffers events in file
Agent1.channels.filechannel.type = file
Agent1.channels.filechannel.checkpointDir = /home/dev/projects/logs/checkpoint/
Agent1.channels.filechannel.dataDirs = /home/dev/projects/logs/data