question Can I create multiple Flume processes on same host? in Archives of Support Questions (Read Only)

Can I create multiple Flume processes on same host?

roman_sydorov1 — Thu, 14 Jan 2016 22:54:30 GMT

Can I run different version of Flume with different classpath on same host?

Re: Can I create multiple Flume processes on same host?

nsabharwal — Thu, 14 Jan 2016 22:58:28 GMT

@Roman Sydorov

Please see this "

It is possible to run multiple Flume agents on the same host

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/installing_flume.html

Re: Can I create multiple Flume processes on same host?

pminovic — Fri, 15 Jan 2016 01:22:08 GMT

@Roman Sydorov I haven't tried but I don't see any reason why not. You can start each Flume version with the command below, just create separate conf directories and set respective classpaths in flume-env.sh. Each one can run multiple agents and each agent will run as a separate process. However, from Ambari you can start and manage only one Flume version specified by default in /etc/flume/conf.

/usr/hdp/current/flume-server/bin/flume-ng agent -c /etc/flume/conf -f /etc/flume/conf/flume.conf -n agent

Re: Can I create multiple Flume processes on same host?

bluesmix — Sat, 16 Jan 2016 04:03:06 GMT

Ambari run each agent in new process. It split single configuration (on ambari screen) to multiple folders (per agent) on the node

Re: Can I create multiple Flume processes on same host?

nsabharwal — Sun, 17 Jan 2016 08:37:12 GMT

@Roman Sydorov

It may not be possible to manage mulitple versions with ambari.

Starting an agent

An agent is started using a shell script called flume-ng which is located in the bin directory of the Flume distribution. You need to specify the agent name, the config directory, and the config file on the command line:

$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template

Now the agent will start running source and sinks configured in the given properties file.

Re: Can I create multiple Flume processes on same host?

pminovic — Sun, 17 Jan 2016 10:21:07 GMT

@Michael M thanks for clarification. @Roman Sydorov Please see my edited answer above.