Hi Team,
I need to put log info from system,hadoop logs in hdfs in same machine
Do we specify multiple sources of flume agent in same machine.
The sample conf file i created is :
# list the sources, sinks and channels in the agent
agent_foo.sources = avro-AppSrv-source1 exec-tail-source2
agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
agent_foo.channels = mem-channel-1 mem-channel-2
# flow #1 configuration
agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1
agent_foo.sources.avro-collection-source1.type = memory
agent_foo.sources.avro-collection-source1.command = tail -F /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-resourcemanager-sri.test.com.log
# flow #2 configuration
agent_foo.sources.exec-tail-source2.channels = mem-channel-2
agent_foo.sources.exec-tail-source2.type = memory
agent_foo.sources.exec-tail-source2.command = tail -F /var/log/messages
# Define a sink that outputs to hdfs dir.
agent_foo.sources.hdfs-sink.channel = mem-channel-1 mem-channel-2
agent_foo.sources.hdfs-sink.type = hdfs
agent_foo.sources.hdfs-sink.hdfs.path = /yarn
It is not working for me.Flume agent is struck with these:
15/09/14 11:08:07 INFO conf.FlumeConfiguration: Added sinks: hdfs-Cluster1-sink1 avro-forward-sink2 Agent: agent_foo
15/09/14 11:08:07 WARN conf.FlumeConfiguration: Agent configuration for 'agent_foo' does not contain any valid channels. Marking it as invalid.
15/09/14 11:08:07 WARN conf.FlumeConfiguration: Agent configuration invalid for agent 'agent_foo'. It will be removed.
15/09/14 11:08:07 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: []
15/09/14 11:08:07 WARN node.AbstractConfigurationProvider: No configuration found for this host:agent
15/09/14 11:08:07 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
Please suggest a solution. ...Sri
agent_foo.sources.hdfs-sink.hdfs.fileType = TEXT
