09-03-2014 02:36 AM
I’m using this flume-snmp-source plugin  to query a managed host with the following logic:
agent.sources.source1.type = org.apache.flume.source.SNMPQuerySource
agent.sources.source1.host = 220.127.116.11
agent.sources.source1.port = 161
agent.sources.source1.delay = 30
agent.sources.source1.oid1 = 18.104.22.168.4.1.2000.1.2.5.1.3
agent.sources.source1.oid2 = 22.214.171.124.4.1.2000.1.2.5.1.7
agent.sources.source1.oid3 = 126.96.36.199.4.1.2000.1.2.5.1.9
agent.sources.source1.oid4 = 188.8.131.52.4.1.2000.1.2.5.1.10
agent.sources.source1.oid5 = 184.108.40.206.4.1.2000.1.2.5.1.12
agent.sources.source1.oid6 = 220.127.116.11.4.1.2000.1.2.5.1.13
agent.sources.source1.oidN = N.N.N.N …
The plugin is a PollableSource source and is quering every “source1.delay” seconds to the managed host “source1.host”. The message passed to the Flume channel is created with this format:
“current date, ip managed device, oid1 answer, oid2 anwswer, …., oidN answer”
The query is made using SNMP GETBULK for performance reasons.
The plugin works fine (the development is in alpha stage), however I’m going ahead with more robust tests, at this point I have the following question related with Flume scaling:
I have to query +1K managed devices with the same snmp query, so I have to created a “source” entry for each host to query in the flume.conf file. This is the correct way to do that, to maintain a huge “flume.conf” file with thousand of entries? Otherwise is there a better strategy for this big scale problems?
Thanks a lot!
09-04-2014 08:37 AM
My first approach would be to create a multi-layer topology where each agent takes control of a subset of that thousands of sources, don't it?
Thus, I'd need a conf mgmt server (i.e. Puppet) to automatize the configuration on these hundreds of agents.