question Re: Flume with oozie in Archives of Support Questions (Read Only)

Flume with oozie

vamsi123 — Tue, 27 Dec 2016 19:52:45 GMT

We are using Flume to get the data into HDFS.After that we are running pig, hive for data transformation.Not sure how to trigger flume from oozie?

Re: Flume with oozie

mjohansson — Tue, 27 Dec 2016 21:52:01 GMT

Hi @vamsi valiveti,

Oozie is a scheduler and Flume is not working on a schedule basis instead Flume is treating the data when it receives it. So you use teh Flume configuration to tell for example that each time there is a file in a certain directory Flume will put it in hdfs (if you use the spooldir source) and so on.

/Best regards, Mats

Re: Flume with oozie

vamsi123 — Tue, 27 Dec 2016 23:19:50 GMT

a)I am starting flume agent using below command.In production how we will trigger this command currently I am running manually on unix command prompt and also i want to create dependeny with hive?

b)can i place below command in unix shell script and call it in shell action in oozie?

flume-ng agent --conf $FLUME_CONF_DIR --conf-file $FLUME_CONF_DIR/flume.conf --name Agent7

Re: Flume with oozie

vamsi123 — Thu, 29 Dec 2016 20:09:29 GMT

Hi @Mats Johansson

Any input on my clarifications

Re: Flume with oozie

bluesmix — Thu, 29 Dec 2016 22:22:33 GMT

@vamsi valiveti you can trigger Flume from the oozie shell action. However pay attention that action will be executed on random cluster node, so all your nodes should have Flume installed. Also you will need to somehow control the agents after that, and if you have >10 nodes it became a problem.. That's why is not common scenario of flume usage.

I'd say the good approach is to keep Flume running all the time. And schedule oozie jobs to process the data whenever you need.

Re: Flume with oozie

vamsi123 — Thu, 29 Dec 2016 22:28:17 GMT

HI @Michael M

Thanks alot for your time.one small clarification

You mentioned good approach is to keep Flume running all the time. And schedule oozie jobs to process the data whenever you need.

clarification 1:-

How to keep Flume running all the time?currently i am using below command on my gateway node.

flume-ng agent --conf $FLUME_CONF_DIR --conf-file $FLUME_CONF_DIR/flume.conf --name Agent7

Re: Flume with oozie

bluesmix — Thu, 29 Dec 2016 22:40:18 GMT

@vamsi valiveti the easiest way is to detach shell from the command using nohup:

nohup <my_command> &

Another option is to create flume init.d service script. I've posted some example script here (search for "Setup flume agent auto startup" on the page), and run the flume as a service.

And third option is to use Ambari to control the agents.

Re: Flume with oozie

vamsi123 — Thu, 29 Dec 2016 23:07:20 GMT

HI @Michael M

For first option:-

In production can I place below command in shell script and schedule that script using crontab so that it will run the Flume will run continuously since In production environment we are not allowed to run any command manually on gateway node.Please correct me if i am wrong?

nohup <my_command> &

Re: Flume with oozie

bluesmix — Thu, 29 Dec 2016 23:51:23 GMT

@vamsi valiveti it could be the option, right.

But for production usage i'd think additionally about how to stop the agents and how to monitor the agent. From my experience init.d service script + ganglia monitoring is a best option.

It allows you to run/stop agents easily with the commands like: /etc/init.d/flume "agent" stop/start. And ganglia provides a nice web interface for the monitoring.