question Re: How to configure Flume on the Hortonworks Sandbox? in Archives of Support Questions (Read Only)

How to configure Flume on the Hortonworks Sandbox?

cwarren — Sat, 17 Sep 2016 06:04:21 GMT

I am preparing the Hortonworks Certification Exam and need assistance configuring Flume on the hortonworks sandbox.

Essentially all I want to do is practice/complete the following tasks from the exam objectives:

Given a Flume configuration file, start a Flume agent
Given a configured sink and source, configure a Flume memory channel with a specified capacity

I see how to start a Flume agent and configure a memory channel etc from the user guide, but you cannot do either of those without providing a configuration file. Unfortunately, I cannot find documentation/tutorials/assistance anywhere on how to set up the configuration file on the Sandbox. I know what needs to go in the configuration file from the user guide, but I can't seem to access the Flume root folder from the system shell so that I can edit the configuration file - thus making it impossible to accomplish the above task.

Does anyone know how this can be done - or, if I am missing something obvious? Or - if anyone knows any good walk through that don't assume prior knowledge of Flume - that would be wonderful.

Re: How to configure Flume on the Hortonworks Sandbox?

ravichandrac65 — Sat, 17 Sep 2016 09:32:12 GMT

Flume configuration file is in

/usr/hdp/current/flume-server/conf

look for flume.conf in the folder and add your requirements to it.

After creating your configuration file use below command to start flume. You have to be in flume home directory to execute this command that is /usr/hdp/current/flume-server

bin/flume-ng agent --conf conf --conf-file flume.conf --name a1 -Dflume.root.logger=INFO,console

For more details refer

https://flume.apache.org/FlumeUserGuide.html

Re: How to configure Flume on the Hortonworks Sandbox?

cwarren — Wed, 21 Sep 2016 03:17:58 GMT

Thanks @R c - this is perfect. My only issue is on the Hortonworks Sandbox on Azure- there doesn't seem to be a /user/hdp/current/flume-server/conf directory. Is this an error or an accidental deletion possible?

Re: How to configure Flume on the Hortonworks Sandbox?

ravichandrac65 — Wed, 21 Sep 2016 03:47:22 GMT

It is not /user/hdp it is /usr/hdp/current or try to cd /usr/hdp/2.4.xxx/flume/ and look for a version number. This location is on local file system not on hdfs. As you are using sandbox it should be the same location.