- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Flume without agents on web server
- Labels:
-
Apache Flume
-
Apache Hadoop
-
HDFS
Created on ‎05-11-2016 01:52 PM - edited ‎09-16-2022 03:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am new to Flume and I want to use Flume to push the web server and application server logs into HDFS (these servers not part of Hadoop cluster). Is it mandatory to install Flume agent (client) on these servers? Can I pull logs from these servers to HDFS without installing Flume agents on them?
Created ‎05-12-2016 07:11 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you run the standalone flume agents, they could use spooldir or the new taildir source (in flume for CDH5.7) to monitor the files and forward via avro to flume agents within your cluster.
-pd
Created ‎05-25-2016 10:38 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎05-12-2016 07:11 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you run the standalone flume agents, they could use spooldir or the new taildir source (in flume for CDH5.7) to monitor the files and forward via avro to flume agents within your cluster.
-pd
Created ‎05-12-2016 09:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
-pd
Created ‎05-16-2016 09:35 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you for detail solution.
I am also wondering , how I can trigger Flume agent on cluster from Edge node (edge node not passwordless ssh to hadoop cluster) as I dont have permission to log in hadoop cluster.
Created ‎05-24-2016 08:29 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎05-25-2016 10:09 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am executing Flumge-ng agnet command on Edge node.
As you already explained in another post, I need to run Flume as service on Edge node to start / stop flume agent.
Created ‎05-25-2016 10:38 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
