Do i need to install flume agent in dedicated servers or datanodes.

[#1]

1. can i install flume agents in dedicated servers or is it ok to install in datanodes .

2. If it is dedicated servers, then how many flume agents in one server

[#2]

You can install them on the edge/utility nodes. Not sure about datanodes, as they could get busy. Probably acceptable in a smaller cluster.

The number of agents per server depends on the volume really. Unless you share details, hard to suggest a solution (or an alternative).

[#3 · Accepted solution]

In the document "http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/installing_flume.html", it states "Hortonworks recommends that administrators not install Flume agents on any node in a Hadoop cluster." That is a really subtle (and hard to notice!) way of saying to put Flume on dedicated servers.

As noted above, in a smaller cluster you can get away with putting them on other nodes. A lot of this depends on the volume of data being processed by Flume and what else if running on the host.

There is also some good info on flume resource at https://cwiki.apache.org/confluence/display/FLUME/Flume's+Memory+Consumption.