Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

multiple sources of flume agent

avatar
Contributor

Hi Team,

 

I need to put log info from system,hadoop logs in hdfs in same machine

Do we specify multiple sources of flume agent in same machine.

 

The sample conf file i created is :

 

# list the sources, sinks and channels in the agent
agent_foo.sources = avro-AppSrv-source1 exec-tail-source2
agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
agent_foo.channels = mem-channel-1 mem-channel-2

# flow #1 configuration
agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1
agent_foo.sources.avro-collection-source1.type = memory
agent_foo.sources.avro-collection-source1.command = tail -F /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-resourcemanager-sri.test.com.log

# flow #2 configuration
agent_foo.sources.exec-tail-source2.channels = mem-channel-2
agent_foo.sources.exec-tail-source2.type = memory
agent_foo.sources.exec-tail-source2.command = tail -F /var/log/messages

# Define a sink that outputs to hdfs dir.
agent_foo.sources.hdfs-sink.channel = mem-channel-1 mem-channel-2
agent_foo.sources.hdfs-sink.type = hdfs
agent_foo.sources.hdfs-sink.hdfs.path = /yarn

 

It is not working for me.Flume agent is struck with these:

 

15/09/14 11:08:07 INFO conf.FlumeConfiguration: Added sinks: hdfs-Cluster1-sink1 avro-forward-sink2 Agent: agent_foo
15/09/14 11:08:07 WARN conf.FlumeConfiguration: Agent configuration for 'agent_foo' does not contain any valid channels. Marking it as invalid.
15/09/14 11:08:07 WARN conf.FlumeConfiguration: Agent configuration invalid for agent 'agent_foo'. It will be removed.
15/09/14 11:08:07 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: []
15/09/14 11:08:07 WARN node.AbstractConfigurationProvider: No configuration found for this host:agent
15/09/14 11:08:07 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

 

Please suggest a solution. ...Sri

agent_foo.sources.hdfs-sink.hdfs.fileType = TEXT

1 ACCEPTED SOLUTION

avatar
Contributor

Hi Harish,

 

We cleared all things and sink is working now.

 

Sridhar

View solution in original post

10 REPLIES 10

avatar
Champion

there is typo in the configuration . 

agent.sinks.agent-sink.channels = agent-chan 

 

to 

 

agent.sinks.agent-sink.channel = agent-chan