Support Questions
Find answers, ask questions, and share your expertise

How to configue multiple different flume agent on different node through ambari?

Explorer

Using ambari not the command line

7 REPLIES 7

Re: How to configue multiple different flume agent on different node through ambari?

You can run multiple flume configs on different hosts using Ambari Config Groups. ( Top of the screen Add Group).

You can run multiple flows in the same flume agent ( simply add multiple flows to the config using different names)

Edit: Thanks to @Michael M for correcting me, you can actually have multiple agents in one node. Ambari now finds all the agents in the config file and provides a list of all of them. With the ability to stop/start them one by one. And I checked that a change to the config will be reflected in the restarted service. Nice!

5053-screen-shot-2016-06-16-at-120139-pm.png

Re: How to configue multiple different flume agent on different node through ambari?

Expert Contributor

@Benjamin Leonhardi, just a small comment:

You CAN run multiple agents on the same host through ambari (at least as of hdp2.3).

Just put all the agents config together, like

agent1.sources..
agent1.sinks..
agent1.channels
...
agent2.sources.
agent2.sinks.
agent2.channels

It will create separate folder for each agent on the host FS

Re: How to configue multiple different flume agent on different node through ambari?

Interesting I was wrong. I thought that was multiple flows running in the same JVM(agent). Now my question would be are these agents all running in the same JVM or is ambari starting one agent/JVM for each of the source keywords.

In any case if you manage it in ambari you need to manage them together I think thats the biggest downside of managing them in ambari. But thanks for correcting me.

Re: How to configue multiple different flume agent on different node through ambari?

Expert Contributor

it will be different agent/jvm. And you will be able to control them separately (start/stop) from ambari. Whenever you're restarting ambari flume service it will recreate the agents (config and .sh script) on local FS

Re: How to configue multiple different flume agent on different node through ambari?

@Michael M need to try it I am pretty sure you cannot start multiple agents on the same node separately from ambari. Let me check it out.

Re: How to configue multiple different flume agent on different node through ambari?

@Michael M Damn you are right. Since when is that feature in? How cool is that. Thanks for correcting me.

5052-screen-shot-2016-06-16-at-120139-pm.png

Re: How to configue multiple different flume agent on different node through ambari?

Expert Contributor

It was there from Ambari 2.0, not sure about earlier versions. There is still not fixed issue preventing us to use it normally:(