Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Store files in hdfs based on keyword

Store files in hdfs based on keyword

Explorer

I am working on flume and i am using spooldir to save files in hdfs but i want to know if i can create my flume conf so that i can store values in my hdfs until i find that keyword.

please help

2 REPLIES 2

Re: Store files in hdfs based on keyword

Master Guru
Apologies but can you rephrase your question, and/or perhaps add an example illustration of what you are trying to achieve?

Its unclear what you're wanting to do - are you looking to stop the flume collection upon some condition?

P.s. Keep in mind that Flume is designed as an event collector, not a file uploader. While Spool sources are available in it, if your first approach leads you to it, you're likely using the wrong tool for the job.

Re: Store files in hdfs based on keyword

Explorer

i have a file where certain words are repeating

e:g: MSH,asa^,hhjs,jack

       OBR,asss,,1019942

       OBX,Hem^dd,,1012193

       MSH,dfdf,9911

 

what i want is that when i put a file using flume spooldir i want to read the contents of file and

store values untill i reach another "MSH". Is it possible using flume ?