How to configure Flume on the Hortonworks Sandbox?

[#1]

I am preparing the Hortonworks Certification Exam and need assistance configuring Flume on the hortonworks sandbox.

Essentially all I want to do is practice/complete the following tasks from the exam objectives:

• Given a Flume configuration file, start a Flume agent
• Given a configured sink and source, configure a Flume memory channel with a specified capacity

I see how to start a Flume agent and configure a memory channel etc from the user guide, but you cannot do either of those without providing a configuration file. Unfortunately, I cannot find documentation/tutorials/assistance anywhere on how to set up the configuration file on the Sandbox. I know what needs to go in the configuration file from the user guide, but I can't seem to access the Flume root folder from the system shell so that I can edit the configuration file - thus making it impossible to accomplish the above task.

Does anyone know how this can be done - or, if I am missing something obvious? Or - if anyone knows any good walk through that don't assume prior knowledge of Flume - that would be wonderful.

[#2 · Accepted solution]

Flume configuration file is in

/usr/hdp/current/flume-server/conf

look for flume.conf in the folder and add your requirements to it.

After creating your configuration file use below command to start flume. You have to be in flume home directory to execute this command that is /usr/hdp/current/flume-server

bin/flume-ng agent --conf conf --conf-file flume.conf --name a1 -Dflume.root.logger=INFO,console

For more details refer

https://flume.apache.org/FlumeUserGuide.html

[#3]

Thanks @R c - this is perfect. My only issue is on the Hortonworks Sandbox on Azure- there doesn't seem to be a /user/hdp/current/flume-server/conf directory. Is this an error or an accidental deletion possible?

[#4]

It is not /user/hdp it is /usr/hdp/current or try to cd /usr/hdp/2.4.xxx/flume/ and look for a version number. This location is on local file system not on hdfs. As you are using sandbox it should be the same location.