I need to find out what the best practice is for running a
set of flume agents in production. All the answers I find dance around the
issue. I am clear that when setting up in Ambari, you can create a number of
config groups for Flume, and each agent needs to be concatenated into the
flume.conf for that group. So, each agent runs 1 instance on each host
associated with the configuration group. At this point, you can see and restart individual agents
through Ambari. However (and here’s the problem), if you make a change to any
of the agents configuration or add a new one then you need to restart ALL of
the agents in that group for the change to take effect! Not acceptable in my case
where I have 4 apps running 2 or 3 agents each. It certainly does not seem to be acceptable to have to restart all applications flume agents whenever a change is made! So, am I missing something or are large enterprises simply
using shell scripts to start the agent on each host? If they are using script, then what
is being used for monitoring and auto-restart?
... View more
