Welcome to the Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

flume+hdfs append

avatar
New Contributor

There is a tool called Fluentd that's similar to Flume but it doesnt roll files but appends data to an existing file instead, is there a way to configure Flume so that it does the same thing?

I've run into several similar topics during googling and the answers were that Flume doesnt support appending, but they're all at least one year old, so I thought maybe Flume has been improved since then, or maybe someone wrote a custom sink (what I'm actually going to do) or something. 

 

What I'm trying to achieve is to stream logs into hdfs and get them available for quering with Impala immediately. As far as I understand, Cloudera Search is developed exactly for that, and I will most probably migrate to it eventually, but now I dont have time for migrating and just want to make Flume+Impala variant work, if that's at all possible.

Any clues will be appreciated.

Who agreed with this topic