Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume without agents on web server

SOLVED Go to solution

Flume without agents on web server

Contributor

I am new to Flume and I want to use Flume to push the web server and application server logs into HDFS (these servers not part of Hadoop cluster). Is it mandatory to install Flume agent (client) on these servers? Can I pull logs from these servers to HDFS without installing Flume agents on them?

 

2 ACCEPTED SOLUTIONS

Accepted Solutions
Highlighted

Re: Flume without agents on web server

Super Collaborator
You need some method to forward the logs to the flume agents. You could do something like rsyslog with the imfile input module (http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html ) to forward to a syslog source on the flume agents, or you could just install the standalone flume agent ( without the rest of the CDH) via rpms or tarball: http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download.html

If you run the standalone flume agents, they could use spooldir or the new taildir source (in flume for CDH5.7) to monitor the files and forward via avro to flume agents within your cluster.

-pd

Re: Flume without agents on web server

Super Collaborator
This documentation goes over stopping and starting flume when not using Cloudera Manager. This assumes you are running packages and not parcels on this edge node: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_flume_run.html
6 REPLIES 6
Highlighted

Re: Flume without agents on web server

Super Collaborator
You need some method to forward the logs to the flume agents. You could do something like rsyslog with the imfile input module (http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html ) to forward to a syslog source on the flume agents, or you could just install the standalone flume agent ( without the rest of the CDH) via rpms or tarball: http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download.html

If you run the standalone flume agents, they could use spooldir or the new taildir source (in flume for CDH5.7) to monitor the files and forward via avro to flume agents within your cluster.

-pd

Re: Flume without agents on web server

Super Collaborator

Re: Flume without agents on web server

Contributor

Thank you for detail solution.

 

I am also wondering , how I can trigger Flume agent on cluster from Edge node (edge node not passwordless ssh to hadoop cluster) as I dont have permission to log in hadoop cluster.

Re: Flume without agents on web server

Super Collaborator
Please define what you mean by "Trigger Flume agent". Are you referring to starting the flume agent, or being able to deliver events to hdfs?

Re: Flume without agents on web server

Contributor

I am executing Flumge-ng agnet command on Edge node.

As you already explained in another post, I need to run Flume as service on Edge node to start / stop flume agent.

Re: Flume without agents on web server

Super Collaborator
This documentation goes over stopping and starting flume when not using Cloudera Manager. This assumes you are running packages and not parcels on this edge node: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_flume_run.html