Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

flume+hdfs append

Highlighted

flume+hdfs append

New Contributor

There is a tool called Fluentd that's similar to Flume but it doesnt roll files but appends data to an existing file instead, is there a way to configure Flume so that it does the same thing?

I've run into several similar topics during googling and the answers were that Flume doesnt support appending, but they're all at least one year old, so I thought maybe Flume has been improved since then, or maybe someone wrote a custom sink (what I'm actually going to do) or something. 

 

What I'm trying to achieve is to stream logs into hdfs and get them available for quering with Impala immediately. As far as I understand, Cloudera Search is developed exactly for that, and I will most probably migrate to it eventually, but now I dont have time for migrating and just want to make Flume+Impala variant work, if that's at all possible.

Any clues will be appreciated.

1 REPLY 1

Re: flume+hdfs append

Rising Star
As far as I can tell, flume still does not enable you to append to files on HDFS.

If you are looking for a solution that allows you to easiliy ingest and append data, I recommend NiFi.