question Do i need to install flume agent in dedicated servers or datanodes. in Support Questions

Do i need to install flume agent in dedicated servers or datanodes.

vdysagar1 — Wed, 11 May 2016 07:40:13 GMT

1. can i install flume agents in dedicated servers or is it ok to install in datanodes .

2. If it is dedicated servers, then how many flume agents in one server

Re: Do i need to install flume agent in dedicated servers or datanodes.

andrewg — Wed, 11 May 2016 19:51:19 GMT

You can install them on the edge/utility nodes. Not sure about datanodes, as they could get busy. Probably acceptable in a smaller cluster.

The number of agents per server depends on the volume really. Unless you share details, hard to suggest a solution (or an alternative).

Re: Do i need to install flume agent in dedicated servers or datanodes.

Jim_B — Wed, 08 Jun 2016 00:42:56 GMT

In the document "http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/installing_flume.html", it states "Hortonworks recommends that administrators not install Flume agents on any node in a Hadoop cluster." That is a really subtle (and hard to notice!) way of saying to put Flume on dedicated servers.

As noted above, in a smaller cluster you can get away with putting them on other nodes. A lot of this depends on the volume of data being processed by Flume and what else if running on the host.

There is also some good info on flume resource at https://cwiki.apache.org/confluence/display/FLUME/Flume's+Memory+Consumption.