Reply
Highlighted
Explorer
Posts: 16
Registered: ‎05-12-2016

Store files in hdfs based on keyword

I am working on flume and i am using spooldir to save files in hdfs but i want to know if i can create my flume conf so that i can store values in my hdfs until i find that keyword.

please help

Posts: 1,903
Kudos: 435
Solutions: 307
Registered: ‎07-31-2013

Re: Store files in hdfs based on keyword

Apologies but can you rephrase your question, and/or perhaps add an example illustration of what you are trying to achieve?

Its unclear what you're wanting to do - are you looking to stop the flume collection upon some condition?

P.s. Keep in mind that Flume is designed as an event collector, not a file uploader. While Spool sources are available in it, if your first approach leads you to it, you're likely using the wrong tool for the job.
Explorer
Posts: 16
Registered: ‎05-12-2016

Re: Store files in hdfs based on keyword

i have a file where certain words are repeating

e:g: MSH,asa^,hhjs,jack

       OBR,asss,,1019942

       OBX,Hem^dd,,1012193

       MSH,dfdf,9911

 

what i want is that when i put a file using flume spooldir i want to read the contents of file and

store values untill i reach another "MSH". Is it possible using flume ?